Add bilateralization tool (project/newpathways-trade)

[junukitashepard]

Overview

The merged code allows users to represent trade bilaterally for specified commodities. By default, the tool bilateralizes existing fuel trade (e.g., oil_exp).

The bilateralization tool, bilateralize, is generalized for any traded commodity, whether that is a fuel (e.g., LNG), or a material (e.g., steel). It also explicitly represents bilateral trade “flows”, or how a fuel/commodity is transported from exporter to importer. These flow technologies are user defined and flexible; the most common are pipelines (e.g., gas pipelines), maritime shipping (e.g., LNG tanker), and transmission lines.

See https://github.com/iiasa/message-ix-models/blob/project/newpathways-trade/doc/api/tools-bilateralize.rst for more details on implementation.

How to review

Note: All scripts are additions (no existing scripts are amended). Therefore, I would request reviewing code directly rather than individual commits (of which there are >200).

These review steps are also outlined in https://github.com/iiasa/message-ix-models/blob/project/newpathways-trade/message_ix_models/tools/bilateralize/workflow.py

1. The function prepare_edit_files() should produce untracked files in commodity-specific directories in data/bilateralize/edit_files and also transfer required parameter files to data/bilateralize/bare_files
2. The function bare_to_scenario() should produce a dictionary of required parameter updates for bilateralization for all specified trade commodities in config.yaml.
3. The function load_and_solve() should be able to be used to update SSP base scenarios
4. The documentation should make sense even though it is incomplete. It is currently conceptual and still needs function documentation.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.

[codecov]

Codecov Report

❌ Patch coverage is 60.31314% with 583 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.5%. Comparing base (3ebe7a9) to head (973ea5f).
⚠️ Report is 419 commits behind head on main.

Files with missing lines	Patch %	Lines
...odels/tools/bilateralize/historical_calibration.py	26.6%	234 Missing ⚠️
message_ix_models/tools/bilateralize/pull_gem.py	18.0%	127 Missing ⚠️
...age_ix_models/tools/bilateralize/load_and_solve.py	57.9%	66 Missing ⚠️
...e_ix_models/tools/bilateralize/bare_to_scenario.py	68.0%	52 Missing ⚠️
..._models/tools/bilateralize/mariteam_calibration.py	20.3%	47 Missing ⚠️
...ssage_ix_models/tools/bilateralize/prepare_edit.py	89.7%	43 Missing ⚠️
...ix_models/tools/bilateralize/calculate_distance.py	80.0%	9 Missing ⚠️
message_ix_models/tools/bilateralize/utils.py	90.3%	3 Missing ⚠️
.../tests/tools/bilateralize/test_bare_to_scenario.py	91.6%	1 Missing ⚠️
...ts/tools/bilateralize/test_mariteam_calibration.py	90.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #438     +/-   ##
=======================================
- Coverage   70.1%   69.5%   -0.6%     
=======================================
  Files        287     300     +13     
  Lines      23129   24598   +1469     
=======================================
+ Hits       16218   17104    +886     
- Misses      6911    7494    +583     

Files with missing lines	Coverage Δ
.../tools/bilateralize/test_historical_calibration.py	100.0% <100.0%> (ø)
...x_models/tests/tools/bilateralize/test_pull_gem.py	100.0% <100.0%> (ø)
message_ix_models/tests/tools/test_bilateralize.py	100.0% <100.0%> (ø)
.../tests/tools/bilateralize/test_bare_to_scenario.py	91.6% <91.6%> (ø)
...ts/tools/bilateralize/test_mariteam_calibration.py	90.0% <90.0%> (ø)
message_ix_models/tools/bilateralize/utils.py	90.3% <90.3%> (ø)
...ix_models/tools/bilateralize/calculate_distance.py	80.0% <80.0%> (ø)
...ssage_ix_models/tools/bilateralize/prepare_edit.py	89.7% <89.7%> (ø)
..._models/tools/bilateralize/mariteam_calibration.py	20.3% <20.3%> (ø)
...e_ix_models/tools/bilateralize/bare_to_scenario.py	68.0% <68.0%> (ø)
... and 3 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[khaeru]

Thanks for the work here! In order to merge the following are needed, and addressing these things first will help reviewers focus more squarely on the added contents and functionality.

• The "PR checklist" in the description should be all complete ✅ or struck out with comments; see here. Per the individual items:
  • "Continuous integration checks all ✅" —I see that there are a few, unrelated tests ERRORing on the "pytest" workflow jobs, but I am also seeing the same errors at Transport improvements for 2025-W39 #430. So I'll address those errors in that PR or another one, and then by rebasing this branch.
  • "Add or expand tests; coverage checks both ✅" —I think this is artificially low because the tests functions are incorrectly decorated and thus do not run. See my comment inline.
  • "Add, expand, or update documentation" —this seems to be done, right? If so, the box can be checked. If not (e.g. there is a plan to still add more documentation in the current PR), please describe.
  • "Update doc/whatsnew" —this could be 1 commit adding 1 line to the file that links to the already-written documentation page. See the blame for the file for examples of prior additions, e.g. ones that added new .tools submodules.
• We prefer a linear history. This can be achieved with git rebase main, which will take all the commits on this branch and rebuild the branch starting from the latest commit on main.
• Note the bullet on commit messages in the "Code style" docs. To change earlier commit messages to meet the style, there are a few approaches, mainly involving git rebase -i:
  • Use reword and edit the commit message.
  • Reorder and squash commits together. When doing this, the commit message (including subject line) for the second, third, etc. commits can be discarded or edited into the first. This is also a strategy for getting rid of commits that might make the history confusing to read.

Many members of the team have experience doing each of these steps and can help by checking work, pair programming, or joining in to do the changes directly, so please reach out on Slack or otherwise.

[junukitashepard]

@khaeru Patch coverage is 51%; however, the scripts that are not covered in testing are those that derive input data based on private data sources. I would of course welcome any input on whether/how to increase this coverage.

[khaeru]

Approved based on:

• Passing checks, notwithstanding codecov/patch as mentioned here.
• Comments by @yiyi1991, which I understand are captured at #460, #462 for follow-up in the next PR.
• Recent rebase and clean-up. In doing this I opened #476 for docs-related comments from November that still need to be addressed.

NB I have not reviewed the code in detail, as the number of commits and amount of added code is too large. But now that the module exists, we can be more particular about its contents as they are improved in future PRs.

I'll merge as soon as the checks pass.