conflicting versions of IEA ETP spread sheets

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

The file /p/projects/rd3mod/inputdata/sources/IEA_ETP/ETP2017_industry_summary.xlsx differs from the ETP2017_industry_summary.xlsx file stored in Zotero.

Notable, the energy figures differ all over the place, by a factor of about 1.8:

• madrat file:

  Total industry final energy consumption (incl. BF, CO and chemical feedstock) (PJ)	2014
  Coal	90374
  Oil	56102
  Natural gas	54622
  Electricity	55091
  Heat	9538
  Biomass	14285
  Waste	456
  Other renewables	61
  Total	280527

• Zotero file:

  Total industry final energy consumption (incl. BF, CO and chemical feedstock) (PJ)	2014
  Coal	49944
  Oil	30536
  Natural gas	29691
  Electricity	30132
  Heat	5164
  Biomass	7905
  Waste	247
  Other renewables	33
  Total	153653

We had an issue like this a couple of years ago with @silviamade, and I'm pretty sure the Zotero data is correct (or at least less wrong than the madrad data), since I matched that up to IEA Energy Balance data as well as I could, and I don't see any way for scraping another 130 EJ together to put into industry.

It appears that this file was uploaded by @MariannaR in 2017 somewhere, but not necessarily here.

$ stat /p/projects/rd3mod/inputdata/sources/IEA_ETP/ETP2017_industry_summary.xlsx
  File: '/p/projects/rd3mod/inputdata/sources/IEA_ETP/ETP2017_industry_summary.xlsx'
  Size: 203299    	Blocks: 448        IO Block: 1048576 regular file
Device: 2ch/44d	Inode: 14132591    Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 3582/ rottoli)   Gid: ( 2662/    rdev)
Access: 2022-03-03 09:37:57.350579286 +0100
Modify: 2017-10-02 17:02:36.000000000 +0200
Change: 2022-02-14 14:22:46.028171402 +0100

Questions:

• Do we have a consensus on the validity of ETP industry data (Zotero vs. madrat)?
• Do we have a consensus on the validity of the other ETP data in madrat (buildings, transport, and the scenario summary)?
• Do we have some way of getting original IEA data again, somehow?
• Who's fixing the data?

[silviamade]

thanks @0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q, I am going to look into this

[silviamade]

I compared the IEA-ETP 2017 World data in 2014 with the IEA Energy Balances, the correct FE for industry should be 153 EJ as reported in the zotero files. I quickly compared the EB with the zotero and madrat IEA-ETP 2017 files also for FE buildings and transport in World 2014. For buildings the FE matches ok across the 3 data sources. For transport, the IEA EB gives me a FE of 95 EJ, whereas both the IEA-ETP 2017 files report 112 EJ. It would be good if the people responsible for the building and transport sector could double check these figurs. It is possible that I am missing something in the way FE is aggregated in the transport sector since I am not an expert. We should try to get hold of the "virgin" dataset because the excel spreadsheets uploaded in zotero are heavily manipulated so they could also contain errors

[MariannaR]

@silviamade thank you for looking into this issue. I have never looked at the ETP data on the cluster, and I am pretty sure I did not modify the files, even if back then I uploaded them. I would also be in favor of re-uploading the original databases - I think I have them still available.

[silviamade]

@MariannaR yes the "culprit" is someone working on buildings, that's where I found the most editing in the files :) If you have the original database could you share with us (me or Michaja) the industry_summary excel? just to check if it is the correct one or not. By the way, do you remember where/how did you get the database? thanks

[MariannaR]

sure, I can share the original files. No I don't remember how I got access to it unfortunately