merge industry subsector structuremappingIO files back into originals

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Running calcOutput("IO", subtype = "output", file = "f04_IO_output.cs4r") once with structuremappingIO_outputs.csv and once with structuremappingIO_outputs_Industry_subsectors.csv as the mapping file yields two results:
f04_IO_output.csv
f04_IO_output_Industry_subsectors.csv

comp <- full_join(
  read_csv(file = '/tmp/f04_IO_output.csv',
                      col_names = c('t', 'regi', 'enty.1', 'enty.2', 'te',
                                    'value_outputs'),
                      col_types = 'iccccd',
                      comment = '*') %>% 
  arrange(t, regi, enty.1, enty.2, te),
  
  read_csv(file = '/tmp/f04_IO_output_Industry_subsectors.csv',
                         col_names = c('t', 'regi', 'enty.1', 'enty.2', 'te',
                                       'value_subsectors'),
                         col_types = 'iccccd',
                         comment = '*') %>% 
  arrange(t, regi, enty.1, enty.2, te),
  
  c('t', 'regi', 'enty.1', 'enty.2', 'te')
)

Differences where outputs and subsectors have different values are limited to

comp %>% 
  filter(value_outputs != value_subsectors) %>% 
  mutate(diff = value_subsectors - value_outputs) %>% 
  distinct(enty.1, enty.2, te)

  enty.1   enty.2   te      
  <chr>    <chr>    <chr>   
1 pecoal   segafos  coalgas 
2 pecoal   sesofos  coaltr  
3 seliqfos fehoi_cs tdhoi_cs

Differences where outputs has values, but subsectors does not:

comp %>% 
  filter(is.na(value_subsectors)) %>% 
  distinct(enty.1, enty.2, te)

   enty.1   enty.2   te      
   <chr>    <chr>    <chr>   
 1 seel     feelcb   tdelb   
 2 seel     feeli    tdeli   
 3 segabio  fegai    tdbiogai
 4 segafos  fegai    tdfosgai
 5 segafos  fegai_cs tdgai_cs
 6 sehe     fehei    tdhei   
 7 seliqbio fehoi    tdbiohoi
 8 seliqfos fehoi    tdfoshoi
 9 sesobio  fesoi    tdbiosoi
10 sesofos  fesoi    tdfossoi

Differences where subsectors has values, but outputs does not:

comp %>% 
  filter(is.na(value_outputs)) %>% 
  distinct(enty.1, enty.2, te) %>% 
  print(n = Inf)

   enty.1   enty.2             te                
   <chr>    <chr>              <chr>             
 1 seel     feel_cement        tdel_cement       
 2 seel     feel_steel         tdel_steel        
 3 seel     feelwlth_chemicals tdelwlth_chemicals
 4 seel     feelwlth_otherInd  tdelwlth_otherInd 
 5 segabio  fega_cement        tdga_cement       
 6 segabio  fega_chemicals     tdga_chemicals    
 7 segabio  fega_otherInd      tdga_otherInd     
 8 segabio  fega_steel         tdga_steel        
 9 segafos  fega_cement        tdga_cement       
10 segafos  fega_chemicals     tdga_chemicals    
11 segafos  fega_otherInd      tdga_otherInd     
12 segafos  fega_steel         tdga_steel        
13 segafos  fegai_cs           tdfosgai_cs       
14 sehe     fehe_otherInd      tdhe_otherInd     
15 seliqbio feli_cement        tdli_cement       
16 seliqbio feli_chemicals     tdli_chemicals    
17 seliqbio feli_otherInd      tdli_otherInd     
18 seliqbio feli_steel         tdli_steel        
19 seliqfos feli_cement        tdli_cement       
20 seliqfos feli_chemicals     tdli_chemicals    
21 seliqfos feli_otherInd      tdli_otherInd     
22 seliqfos feli_steel         tdli_steel        
23 sesobio  feso_cement        tdso_cement       
24 sesobio  feso_chemicals     tdso_chemicals    
25 sesobio  feso_otherInd      tdso_otherInd     
26 sesobio  feso_steel         tdso_steel        
27 sesofos  feso_cement        tdso_cement       
28 sesofos  feso_chemicals     tdso_chemicals    
29 sesofos  feso_otherInd      tdso_otherInd     
30 sesofos  feso_steel         tdso_steel   

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

• add summation of subsector FE use, should provide fe(so|he|ga|h2|he|el)i
• check segafos . fegai_cs . tdfosgai_cs ./. segafos . fegai_cs . tdgai_cs (genuine inconsistency?)
• what is seel . feelcb . tdelb?
• map out coalgas, coaltr, and tdhoi_cs differences

[robinhasse]

* [ ]  what is `seel . feelcb . tdelb`?

I guess I should know but to be honest, I never understood why we have stationary in the model and so I don't know what this is supposed to be. From REMIND code I understand that tdelb is feels -> feelb and not the full t&d from secondary level. Also feelcb would be just the non-heating part of buildings electricity which is incomplete. Sorry for this much cluelessness...

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

It is sufficient to know that it is used somewhere in REMIND.
I guess that Antoine added it after I created structuremappingIO_outputs_Industry_subsectors.csv. If only changes to those files would have been tracked …
Question @robinhasse: is buildings/simple used and operational?

[robinhasse]

is buildings/simple used and operational?

It is the only realisation that is used and operational.
Also, seel.feelcb.tdelb likely comes from @Renato-Rodrigues as there was no differentiation of electricity back when Antoine worked on this.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

It is the only realisation that is used and operational.

Ouch.

[nicobauer]

@robinhasse "stationary" is the aggregate of buildings, industry, services, etc... everything that is not moving. That was a aggregate used many years ago, when we did not want to increase the number of ces-nests too much. As it is: such things keep on living in code after they were abandoned. Anyways...
regarding the feelcb it seems to me that this is what used to be feelb. I see the same entries in structuremappingIO_outputs.csv with feelcb that were used in structuremappingIO_outputs_old.csv but with feelb. That seeme to be a renaming or it is simply a bug.
On the entries in the table I agree with @robertpietzcker about his remakrs about deleting unused mappings.

[robinhasse]

Thanks, Nico. It is not just a renaming. We used to only have feelb (all electricity in buildings). Now it is the sum of feelcb (non-heating), feelrhb (resistive heating) and feelhpb (heat pumps). So I actually think that seel.feelcb.tdelb is wrong. I though that tdels is seel --> feels and than we have the two links to buildings and industry tdeli: feels --> feeli and tdelb: feels --> feelb.

[nicobauer]

@robinhasse did I understand correctly that this increased the number of primary inputs into the overall CES-tree? If so, I recommend to strongly evaluate whether this is really necessary. Every CES-nest and -branch makes the model more difficult to calibrate.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

@nicobauer, you are about two and a half years late with that admonition. And the calibration can deal with that just fine.

[robertpietzcker]

Also feelcb would be just the non-heating part of buildings electricity which is incomplete.

but is there actually a "heat pump electricity" input for 2005 in REMIND? because how would you get such a value if IEA doesn't tell you how electricity for residential splits into the different uses?

or is there some buildings-related adjustment on the IEA numbers that splits the elec going into buildings by a 10%/90% split into heat pump / other elec?

so it seems to me that (after checking it is correct) this part would need to be transferred into a new "structuremapping_IO_merged" - or how is your process proposal for this, Michaja? I would think creating a third "merged" mapping will best allow us to determine if it really produces all the right output when the two old instances are replaced by the new merged one.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

My plan is to recreate the output of the outputs mapping & code with the outputs_Industry_subsectors mapping & code verbatim, except for the instances where they should be extended or are wrong.

[robertpietzcker]

but is there actually a "heat pump electricity" input for 2005 in REMIND? because how would you get such a value if IEA doesn't tell you how electricity for residential splits into the different uses?

could you briefly think about this and confirm/correct, @robinhasse ?

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

but is there actually a "heat pump electricity" input for 2005 in REMIND? because how would you get such a value if IEA doesn't tell you how electricity for residential splits into the different uses?

could you briefly think about this and confirm/correct, @robinhasse ?

1. Irrelevant to this issue.
2. No, there is only feelcb in f04_IO_output.cs4r:

   $ tar -Oxf /p/projects/rd3mod/inputdata/output/rev6.605_62eff8f7_remind.tgz ./f04_IO_output.cs4r | awk -F ',' '$1 !~ /^\*/ && !a[$4] ++ { print $4 }'
   seliqfos
   fehoi
   fesoi
   fesob
   sesofos
   seel
   sehe
   fedie
   fehob
   seliqbio
   fegai
   fegab
   segabio
   fegat
   fepet
   fehoi_cs
   segafos
   sesobio
   feeli
   feel
   feelt
   fehei
   feheb
   fegai_cs
   feelcb
   feelb
   

[robertpietzcker]

1. Irrelevant to this issue.

Hm, I had thought the original question if/whether seel . feelcb . tdelb is something that should be there or something that should in fact be different/be called differently was still relevant, because I did not get a clear idea from Robin's reactions if this is indeed needed or not.

in your list I see feelcb and feelb, and I currently think they should NOT both be outputs from f04_IO_output.cs4r, but only one of the two. (based on the assumption that the IEA data cannot differentiate between the two)

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Hm, I had thought the original question if/whether seel . feelcb . tdelb is something that should be there or something that should in fact be different/be called differently was still relevant, because I did not get a clear idea from Robin's reactions if this is indeed needed or not.

If that is the case, it applies to the output mapping as well. I will just recreate that mapping (where industry is not concerned). So this is not relevant here.

[robertpietzcker]

Ok, I understand.

Is there any further support I can offer for anything in this issue, or is it mainly "finding time" ?

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Nothing right now.

[robertpietzcker]

pushing this up again - is there any progress? if not - what are the chances this advances over the next week, and how can they be increased?

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

This is not going to be finished next week, and the chances of changing that are zero (short of Joshua 10:13).

[robertpietzcker]

thanks for the info. (and I wanted to write "weeks", not "week", but given your reply on the other issue I guess the answer remains the same

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

But to forward this issue in general:

• check segafos . fegai_cs . tdfosgai_cs ./. segafos . fegai_cs . tdgai_cs (genuine inconsistency?)

@robertpietzcker are fehoi_cs and fegai_cs still used?
They do appear in f04_IO_output.cs4r (generated by calcIO()), but in the code they only show up in ./core/sets.gms as part of all_enty [↑], but in no mappings. And that appears to have been the case for the last four years [↑].
So, is there some magical mechanism by which these two have an bearing on the PE-to-SE calibration, or can we just cut them?

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

But to forward this issue in general:

• check segafos . fegai_cs . tdfosgai_cs ./. segafos . fegai_cs . tdgai_cs (genuine inconsistency?)

@robertpietzcker are fehoi_cs and fegai_cs still used? They do appear in f04_IO_output.cs4r (generated by calcIO()), but in the code they only show up in ./core/sets.gms as part of all_enty [↑], but in no mappings. And that appears to have been the case for the last four years [↑]. So, is there some magical mechanism by which these two have an bearing on the PE-to-SE calibration, or can we just cut them?

*poke* @robertpietzcker

[robertpietzcker]

Ha, I somehow didn't notice the poke - sorry for that.

I checked the structuremapping to understand what the _cs means, and it stands for ``_carbonstored` if I am not mistaken.

So I guess this was an old way of trying to calculate what amount of the carbon going into nonenergy is burned in the end.

I would guess @nicobauer was the one who originally implemented this, but I don't know.
My gut feeling is that with the new nonenergy implementation, these two variables are not needed anymore.

but it would be good if all the other people dealing with carbon flows (@nicobauer , @strefler ?) could confirm that they are not using this for anything?

also pinging @Renato-Rodrigues and @fschreyer just to check that you are also not using fehoi_cs and fegai_cs for anything?