Add bilateralization tool (project/newpathways-trade)#438

Merged

khaeru merged 195 commits into

mainfrom

project/newpathways-trade

Feb 13, 2026

Commits on Feb 11, 2026

Initial bilateralize tool and config

junukitashepard
authored and
khaeru
committed
Build and test bare sheets for oil_shipped

junukitashepard
authored and
khaeru
committed
Update .gitignore

junukitashepard
authored and
khaeru
committed

Commits on Feb 12, 2026

Simplify gas pipeline framework

junukitashepard
authored and
khaeru
committed
Add diagnostics and calibration

junukitashepard
authored and
khaeru
committed
Remove scenario run requirement

junukitashepard
authored and
khaeru
committed
Add historical calibration

junukitashepard
authored and
khaeru
committed
Add historical calibration

junukitashepard
authored and
khaeru
committed
Add tracked metrics to config

junukitashepard
authored and
khaeru
committed
Add sankey for diagnostics of bilateral trade

junukitashepard
authored and
khaeru
committed
Link to GEM for historical infrastructure data

junukitashepard
authored and
khaeru
committed
Add historical calibration and option to output gdx (no solve)

junukitashepard
authored and
khaeru
committed
Add flow technologies to full function (bilateral only)

junukitashepard
authored and
khaeru
committed
Run with steel data inputs

junukitashepard
authored and
khaeru
committed
Add flow tec to diagnostics

junukitashepard
authored and
khaeru
committed
Update sankey.py

junukitashepard
authored and
khaeru
committed
Add flows and proxy for 2025 with 2023 activity

junukitashepard
authored and
khaeru
committed
Specify gas pipes in historical calibration

junukitashepard
authored and
khaeru
committed
Make sure base technology gets deleted

junukitashepard
authored and
khaeru
committed
Final global flow constraint commit

junukitashepard
authored and
khaeru
committed
Add MariTEAM and calibrate LNG

junukitashepard
authored and
khaeru
committed
Restructure code between /tools and /project

junukitashepard
authored and
khaeru
committed
Clean up pipeline calibration and GEM connection

junukitashepard
authored and
khaeru
committed
LFS update to LNG Tankers

junukitashepard
authored and
khaeru
committed
Technical lifetime for LNG to 20 years

junukitashepard
authored and
khaeru
committed
Run diagnostics for LNG and pipeline

junukitashepard
authored and
khaeru
committed
Create newpathways-trade.rst

junukitashepard
authored and
khaeru
committed
Set up generate_bare_sheets so default values populated in bare_files

junukitashepard
authored and
khaeru
committed
Refactor and run bilateralize.py

junukitashepard
authored and
khaeru
committed
Update flow formulation in bilateralization

junukitashepard
authored and
khaeru
committed
Add historical_new_capacity for pipelines

junukitashepard
authored and
khaeru
committed
Add year_vtg and begin at 2000 for historical_new_capacity of pipelines

junukitashepard
authored and
khaeru
committed
Rerun edit files with LNG flow tech as input to trade tech

junukitashepard
authored and
khaeru
committed
Remove gas_piped from .gitignore

junukitashepard
authored and
khaeru
committed
Update config_default.yaml

junukitashepard
authored and
khaeru
committed
Add p-drive flexibility to GEM and MariTeam call

junukitashepard
authored and
khaeru
committed
Add project name/config to bare_to_scenario

junukitashepard
authored and
khaeru
committed
Rebuild BACI for historical_calibration

junukitashepard
authored and
khaeru
committed

Commits on Feb 13, 2026

Set up load_and_solve

junukitashepard
authored and
khaeru
committed
Update load_and_solve

junukitashepard
authored and
khaeru
committed
Set up LNG with updated flow framework

junukitashepard
authored and
khaeru
committed
Set config to solve MESSAGE

junukitashepard
authored and
khaeru
committed
Correct foil commodity to fueloil

junukitashepard
authored and
khaeru
committed
Add cost to LNG trade

junukitashepard
authored and
khaeru
committed
Update variable cost for LNG trade

junukitashepard
authored and
khaeru
committed
Remove pydeflate

junukitashepard
authored and
khaeru
committed
Add variable cost for LNG shipping

junukitashepard
authored and
khaeru
committed
Move variable cost to investment cost for LNG shipping

junukitashepard
authored and
khaeru
committed
Add fix cost to LNG shipping

junukitashepard
authored and
khaeru
committed
Add activity constraints to LNG shipping trade technology

junukitashepard
authored and
khaeru
committed
Remove constraint extend tec lifetime for LNG trade tec

junukitashepard
authored and
khaeru
committed
Add growth and initial activity constraints to LNG trade tec

junukitashepard
authored and
khaeru
committed
Add Sankey and Chord functionalities

junukitashepard
authored and
khaeru
committed
Differentiate distances for maritime shipping

junukitashepard
authored and
khaeru
committed
Add LNG tanker historical new capacity, lower LNG trade growth constraints

junukitashepard
authored and
khaeru
committed
Change year_vtg to integer in historical new capacity

junukitashepard
authored and
khaeru
committed
Update config.yaml

junukitashepard
authored and
khaeru
committed
Add trade historical new capacities (not just flow)

junukitashepard
authored and
khaeru
committed
Make year_vtg integer in hist_new_capacity for trade

junukitashepard
authored and
khaeru
committed
Add gas pipeline to RCPA

junukitashepard
authored and
khaeru
committed
Add soft activity bounds to LNG trade tec

junukitashepard
authored and
khaeru
committed
Update var_costs and tec_lt for LNG tankers

junukitashepard
authored and
khaeru
committed
Add cost scenarios

junukitashepard
authored and
khaeru
committed
Add level_cost_activity_soft and abs_cost_activity_soft constraints for LNG, add additional parameter update option to clone_and_update

junukitashepard
authored and
khaeru
committed
Set var_cost to 5 year for tankers, add scenario for NAM shale discount

junukitashepard
authored and
khaeru
committed
Update inv_cost based on higher lifetime Mt-km

junukitashepard
authored and
khaeru
committed
Rerun all scenarios

junukitashepard
authored and
khaeru
committed
Remove technical lifetimes from config.yaml

junukitashepard
authored and
khaeru
committed
Reset broadcasting function in bilateralization

junukitashepard
authored and
khaeru
committed
Add var cost to LNG trade

junukitashepard
authored and
khaeru
committed
Re-add all test scenarios

junukitashepard
authored and
khaeru
committed
Adjust variable costs on LNG

junukitashepard
authored and
khaeru
committed
Update load_and_solve.py

junukitashepard
authored and
khaeru
committed
Add options for flow capacity constraints; debug trade_parameter dictionary

junukitashepard
authored and
khaeru
committed
Add distance script to generalize further regionalization

junukitashepard
authored and
khaeru
committed
Update test scenarios

junukitashepard
authored and
khaeru
committed
Add var_cost to bare by default, add more test scenarios

junukitashepard
authored and
khaeru
committed
Fix pathname in pull_gem

junukitashepard
authored and
khaeru
committed
Fix pathname in mariteam

junukitashepard
authored and
khaeru
committed
Remove max/min on var costs

junukitashepard
authored and
khaeru
committed
Adjust test scenario on var cost

junukitashepard
authored and
khaeru
committed
Set default variable cost to mean instead of min

junukitashepard
authored and
khaeru
committed
Round var cost values

junukitashepard
authored and
khaeru
committed
Comment out test scenarios that were already run

junukitashepard
authored and
khaeru
committed
Add up-to-date test scenarios in diagnostics

junukitashepard
authored and
khaeru
committed
Add steel back; add regionality to bilateral commodities

junukitashepard
authored and
khaeru
committed
Set up liquefaction spreadsheet; only run baseline test scenario (pipelines_LNG)

junukitashepard
authored and
khaeru
committed
Add liquefaction test scenario

junukitashepard
authored and
khaeru
committed
Debug: trade_parameters add

junukitashepard
authored and
khaeru
committed
Add base diagnostics and clean up newpathways project repo

junukitashepard
authored and
khaeru
committed
Update Russian LNG terminal from Novorossiysk to Arctic LNG 2

junukitashepard
authored and
khaeru
committed
Transfer Russina LNG terminal update to bare file

junukitashepard
authored and
khaeru
committed
Update scenario pickle

junukitashepard
authored and
khaeru
committed
Remove steel as input to activity

junukitashepard
authored and
khaeru
committed
Add HHI test scenarios

junukitashepard
authored and
khaeru
committed
Remove reporting tables

junukitashepard
authored and
khaeru
committed
Use path join for hhi_config name

junukitashepard
authored and
khaeru
committed
Set HHI test to WEU only (assume CHN will break)

junukitashepard
authored and
khaeru
committed
Small debug

junukitashepard
authored and
khaeru
committed
Add coal representation

junukitashepard
authored and
khaeru
committed
Finish adding coal, option to remove domestic_coal/gas relation activity

junukitashepard
authored and
khaeru
committed
Set up constraints on coal

junukitashepard
authored and
khaeru
committed
Add shipped crude

junukitashepard
authored and
khaeru
committed
Add crude oil pipelines

junukitashepard
authored and
khaeru
committed
Update GEM calibration for oil (and gas) pipelines

junukitashepard
authored and
khaeru
committed
Set load_and_solve to solve ixmp

junukitashepard
authored and
khaeru
committed
Ensure years are integers in historical_activity

junukitashepard
authored and
khaeru
committed
Remove var cost on crude oil shipping

junukitashepard
authored and
khaeru
committed
Add ethanol shipping

junukitashepard
authored and
khaeru
committed
Add 2025 values

junukitashepard
authored and
khaeru
committed
Update costs to shipped and piped crude

junukitashepard
authored and
khaeru
committed
Add fuel oil trade

junukitashepard
authored and
khaeru
committed
Add costs to fuel oil shipping

junukitashepard
authored and
khaeru
committed
Add fuel oil piped trade

junukitashepard
authored and
khaeru
committed
Add light oil piped and shipped

junukitashepard
authored and
khaeru
committed
Ensure that years in GEM based files are integers

junukitashepard
authored and
khaeru
committed
Light oil should be used for shipping by default

junukitashepard
authored and
khaeru
committed
LNG shipping updated to fueled by light oil

junukitashepard
authored and
khaeru
committed
Remove piped light oil for debugging

junukitashepard
authored and
khaeru
committed
Set load_and_solve to solve (not to_gdx)

junukitashepard
authored and
khaeru
committed
Remove NaN from output for flow_tech loil

junukitashepard
authored and
khaeru
committed
Add 1 to outputs for shipped coal flow

junukitashepard
authored and
khaeru
committed
Add loil_piped

junukitashepard
authored and
khaeru
committed
Add biomass shipping

junukitashepard
authored and
khaeru
committed
Add lh2 trade

junukitashepard
authored and
khaeru
committed
Rename H2 scenario and updated diagnostics

junukitashepard
authored and
khaeru
committed
Set base trade tecnames so that base technologies are removed

junukitashepard
authored and
khaeru
committed
Add fuel oil bunker technology to foil_ship

junukitashepard
authored and
khaeru
committed
Add cost and growth constraints for sensitivities

junukitashepard
authored and
khaeru
committed
Remove constraints for fuel oil, lh2, loil

junukitashepard
authored and
khaeru
committed
Add pickle back to load_and_solve.py

junukitashepard
authored and
khaeru
committed
Debug LH2 configuration

junukitashepard
authored and
khaeru
committed
Test scenario after LH2 debug

junukitashepard
authored and
khaeru
committed
Set constraint parameter removal to be flexible

junukitashepard
authored and
khaeru
committed
Update .gitignore to ignore diagnostic data

junukitashepard
authored and
khaeru
committed
Remove caches for untracked files

junukitashepard
authored and
khaeru
committed
Move configs to technology-specific yamls

junukitashepard
authored and
khaeru
committed
Delete build_sankey.py

junukitashepard
authored and
khaeru
committed
Update configs

junukitashepard
authored and
khaeru
committed
Remove inter_pipe (artifact)

junukitashepard
authored and
khaeru
committed
Add workflow and test

junukitashepard
authored and
khaeru
committed
Allow user specification on scenario updates (rather than config)

junukitashepard
authored and
khaeru
committed
Allow diagnostics for NEWPATHWAYS for all commodities

junukitashepard
authored and
khaeru
committed
Don't allow pipelines from FSU to PAO

junukitashepard
authored and
khaeru
committed
Add information on gas compression along pipelines

junukitashepard
authored and
khaeru
committed
Add scenario flexibility to load_and_solve

junukitashepard
authored and
khaeru
committed
Ensure historical activity loads only if pipeline is specified

junukitashepard
authored and
khaeru
committed
Refactor bilateralize.py

junukitashepard
authored and
khaeru
committed
Remove project/newpathways from branch

junukitashepard
authored and
khaeru
committed
Remove implied optionals

junukitashepard
authored and
khaeru
committed
Split bilateralize into step code

junukitashepard
authored and
khaeru
committed
Reformat to meet mypy, ruff, and testing

junukitashepard
authored and
khaeru
committed
Create documentation tools-bilateralize

junukitashepard
authored and
khaeru
committed
Remove pytest fixtures from test_bilateralize.py

junukitashepard
authored and
khaeru
committed
Update test_bilateralize and associated files

junukitashepard
authored and
khaeru
committed
Add more tests to test_bilateralize

junukitashepard
authored and
khaeru
committed
Reformat prepare_edit for mypy and ruff

junukitashepard
authored and
khaeru
committed
Update test_bilateralize.py to meet mypy

junukitashepard
authored and
khaeru
committed
Satisfy ruff in test_bilateralize

junukitashepard
authored and
khaeru
committed
Update output directory in pull_gem.py

junukitashepard
authored and
khaeru
committed
Point project and config names in subfunctions

junukitashepard
authored and
khaeru
committed
Add tests of .bilat.{historical_calibration,pull_gem}
khaeru
committed
Remove data/bilateralize/node_lists and references

junukitashepard
authored and
khaeru
committed
Add opiton to load and solve scenario directly

junukitashepard
authored and
khaeru
committed
Create __init__.py

junukitashepard
authored and
khaeru
committed
Add bilateralize documentation to doc indices

junukitashepard
authored and
khaeru
committed
Update file locations in pull_gem.py

junukitashepard
authored and
khaeru
committed
Only remove base oil trade if crude oil is called

junukitashepard
authored and
khaeru
committed
Change GEM to csv in P-drive

junukitashepard
authored and
khaeru
committed
Add relation activity for regional exports

junukitashepard
authored and
khaeru
committed
Ensure project/config name specified in build_historical_price

junukitashepard
authored and
khaeru
committed
Update testing and formatting to meet ruff/mypy

junukitashepard
authored and
khaeru
committed
Formatting for ruff

junukitashepard
authored and
khaeru
committed
Change var_cost to fix_cost for trade parameters

junukitashepard
authored and
khaeru
committed
Allow for user to set base trade technologies

junukitashepard
authored and
khaeru
committed
Update utf encoding when reading yaml

junukitashepard
authored and
khaeru
committed
Add technical lifetime as a user configuration

junukitashepard
authored and
khaeru
committed
Use existing util.broadcast in bare_to_scenario

junukitashepard
authored and
khaeru
committed
Update tools-bilateralize.rst

junukitashepard
authored and
khaeru
committed
Update tools-bilateralize.rst

junukitashepard
authored and
khaeru
committed
Debug bilateralize test

junukitashepard
authored and
khaeru
committed
Add flexibility to bare_to_scenario

junukitashepard
authored and
khaeru
committed
Ensure shipped costs transfer from edit to bare

junukitashepard
authored and
khaeru
committed
Add test back

junukitashepard
authored and
khaeru
committed
Updates based on alps workflow

junukitashepard
authored and
khaeru
committed
Ruff formatting

junukitashepard
authored and
khaeru
committed
Add test for bare_scenario

junukitashepard
authored and
khaeru
committed
Add test coverage

junukitashepard
authored and
khaeru
committed
Update test_bare_to_scenario.py

junukitashepard
authored and
khaeru
committed
Set test_calibrate_historical_shipping as xfail

junukitashepard
authored and
khaeru
committed
Reformat for ruff

junukitashepard
authored and
khaeru
committed
Add scgraph to pyproject.toml

junukitashepard
authored and
khaeru
committed
Add test coverage for mariteam_calibration

junukitashepard
authored and
khaeru
committed
Improve doc/tools-bilateralize.rst
khaeru
committed
Add to #438 to doc/whatsnew.rst

junukitashepard
authored and
khaeru
committed