Add .tools.sankey and tutorial

[mabudz]

Create sankey diagram and add tutorial.

The approach here creates the sankey diagram by using the sankey plot function of the pyam package. This function requires a mapping dictionary. The mapping dict is created by using the newly created automatic report message::sankey and a utility function sankey_mapper()map_for_sankey.

This pull request is related to the draft pull request in message_data , which can be removed, since the sankey diagram has been implemented in the pyam package.

Note

The new key is now only added when calling Reporter.add_sankey().

How to review

General approach should be reviewed. Also, I am not sure if the util function sankey_mapper() map_for_sankey() is at the right place.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update release notes.

[CLAassistant]

All committers have signed the CLA.

[glatterf42]

Thanks for this PR :)
Getting the tests to pass is a bit tricky since the API for pyam changed when they went from version 1.9.0 to 2.0.0 (as you might expect) and our code needs to satisfy both versions since we test on Python <= 3.9, which is not supported for pyam 2.0.0. So let me take a closer look, maybe we'd want to use the function that pyam's sankey function wraps around directly, after all.

[glatterf42]

Thanks @daymontas1 for stepping up here, much appreciated :)

You should be able to test the PR as is and see how you like it's usefulness, but there are a few things that I would like to see happen before we merge it:

• Please rebase on main to keep it up to date with current developments.
• Please migrate the main functionality from util/__init__.py to somewhere more appropriate. E.g. report/sankey.py if it's a reporting tool or util/sankey.py if there are more general use cases.
• Please check if we can use plotly directly instead of pyam's wrapper functionality. This should allow us to utilize more design options and avoids some dependency constraints (see above). See some examples and their docs.
• Please confirm that we want to integrate the sankey functionality in the reporting as it currently stands. I'd probably ask at least @khaeru for his opinion.
• Please write a test if possible to confirm that sankey_mapper maps the correct values to the correct keys. We might also need to adapt one or two reporting tests that check the number of reporting steps if this stays in the default reporting workflow.

Please feel free to ask about any of these steps if you need help :)

[gidden]

Hi all, to simplify here, I would simply limit this functionality to pyam > 2 - we can do a version check for this on the fly and raise an error.

[gidden]

I would get rid of this, and simply do the concatenation outside of reporter (e.g., in the jupyter notebook here)

[glatterf42]

I kept this for now as I don't know what the Reporter actually does and it seems useful to just get the dataframe format we need from the Reporter. But please elaborate :)

[daymontas1]

I have created a new file, sankey.py, within the report directory, which extracts Sankey diagram functionality from init.py. Fridolin (@glatterf42), could you please confirm if this is what we want? Additionally, I have updated the westeros_sankey.ipynb file to ensure compatibility with the new sankey.py file in the report directory. I have annotated these changes in the westeros_sankey.ipynb below. Also, in this case, the following line inside the init.py file must be either removed or commented out:
("message::sankey", "concat", "out::pyam", "in::pyam")

[glatterf42]

Thanks, @daymontas1, these changes sound good to me. For the line that has become redundant, please remove it rather than commenting it out, this keeps the code clean.
Could you please push your changes to the branch so that we can take a look and see how the tests are doing?
If you don't have write access to @mabudz's fork, please let us know so that we can figure out a solution :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.6%. Comparing base (a60715a) to head (9926003).
Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #770   +/-   ##
=====================================
  Coverage   95.6%   95.6%           
=====================================
  Files         46      48    +2     
  Lines       4338    4394   +56     
=====================================
+ Hits        4148    4204   +56     
  Misses       190     190           

Files with missing lines	Coverage Δ
message_ix/report/__init__.py	100.0% <100.0%> (ø)
message_ix/tests/test_report.py	100.0% <100.0%> (ø)
message_ix/tests/test_tutorials.py	97.6% <ø> (ø)
message_ix/tests/tools/test_sankey.py	100.0% <100.0%> (ø)
message_ix/tools/sankey.py	100.0% <100.0%> (ø)

[glatterf42]

In today's team meeting, @SiddharthJoshi-Git mentioned that he is fine with merging this PR as is and providing feedback later on. So I'm happy to merge this once the tests pass, which ... seem kind of confused about which versions of Python they should be using. For message_ix, that is still 3.8 and not 3.13, so maybe we'll have to merge #881 first.

[khaeru]

Overall this is a welcome contribution. Several things that I will “request” as changes but actually myself implement on the branch:

1. The additional steps between rep.get("message::sankey") and fig.show() that appear in the tutorial can be added to the Reporter.add_sankey() method and controlled with keyword arguments. This saves the user having to manually re-write these steps for every such diagram they want to create.
2. For (1), but also generally, plotly as an added dependency appears required to use the feature(s) as intended/recommended. So I would make a new group of optional dependencies message_ix[sankey] = [plotly], and then make message_ix[sankey] a recursive dependency in the tutorials group.
3. The tests should be expanded to cover the whole range of usage.
4. The new methods and functions must appear in the documentation; currently from the preview build they do not. (A tutorial does not substitute for API documentation.)
5. The release note line should link directly to the tutorial and to the documentation of the new function(s).

Other comments:

• The history of the branch appears to have some duplicate commits (probably due to failure to rebase/inconsistent merging with and rebasing on main) and commit names. We should thus squash-and-merge once this is ready to go.

• In order to take advantage of the sankey features via pyam, the data appear to be used via the IAMC structure in which "variable" names like "in|secondary|electricity|grid|standard" are constructed by mashing together labels for distinct dimensions, and then these are again deconstructed using .util.sankey.set_start_and_end_index() / get_variable_components(). This is both complicated and fragile: if the user chooses to construct "variable" names differently, it will not work.

  For other purposes we've discussed, like drawing RES diagrams using NetworkX/graphviz/others (FYI @macflo8), it will be simpler to use the data at their full/original resolution and construct node IDs using unique combinations of labels for certain dimensions, like (level, commodity), (technology, mode), or others. This may also end up providing a more direct way to construct Sankey diagrams, but we will see once we get there.