Feat/streamline merge - #888
Draft
FabianHofmann wants to merge 225 commits into
Draft
Conversation
- Add test_streamlined.smk with 4-step workflow (base → clustered → composed → solved) - Support all three foresight modes: overnight, myopic, perfect - Add robust horizon handling for single values and lists - Update test configs with proper temporal and clustering sections - Add comprehensive test configurations for all foresight modes - Add implementation plan and documentation Features: - Config-driven approach (no wildcard expansion) - Dynamic input resolution based on foresight and horizons - Sequential dependencies for myopic/perfect foresight - Unified rule structure across all modes - Robust parameter handling with defaults Tested workflows: - Overnight: 6 jobs, single horizon optimization - Myopic: 10 jobs, sequential horizon optimization with brownfield - Perfect: 8 jobs, multi-period simultaneous optimization Refs: GitHub Discussion PyPSA#1529
- Add Status Update section documenting completed proof of concept - Update timeline to reflect completed Phase 0 (POC) and current focus - Document validated implementation patterns from testing experience - Add robust parameter handling and dynamic input resolution examples - Update success criteria to distinguish completed vs remaining goals - Reorganize open questions to focus on remaining implementation details - Document created test files and artifacts Key achievements validated: - 4-step workflow architecture working across all foresight modes - Config-driven approach with temporal section replacing scenario wildcards - Dynamic dependencies with proper sequential chaining - Robust error handling for missing config sections - Flexible horizon handling for single values and lists Next phase: Implement actual scripts with real PyPSA network operations
- Create scripts/compose_network.py that combines all network building steps
in one big main section without additional function definitions
- Implement network concatenation logic for perfect foresight mode
- Add brownfield constraints for myopic optimization
- Create production rules/compose.smk with dynamic dependencies
- Update Snakefile to include compose rules when streamlined_workflow is enabled
- Add temporal configuration section to default config
- Create example configuration file demonstrating the new workflow
The new workflow follows the 4-step structure:
base → clustered → composed_{horizon} → solved_{horizon}
All configuration is now driven by config sections rather than wildcards,
simplifying the workflow and making it more maintainable.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
CRITICAL FIXES:
- Fix snapshot handling bug: prevent overwriting multi-period snapshots in perfect foresight
- Fix function signature error in load_and_aggregate_powerplants call
- Remove resource path conflict by using existing path provider system
- Simplify overly complex perfect foresight concatenation logic
- Fix parameter access patterns throughout the script
- Improve error handling for missing config sections
RESOURCE PATH UPDATES:
- Update all resource references to use existing path patterns with {clusters} wildcards
- Remove custom resources() function that conflicted with existing system
- Align file paths with existing workflow conventions
ROBUSTNESS IMPROVEMENTS:
- Add proper error handling for missing inputs
- Simplify carbon budget handling to avoid complex logic failures
- Use safer parameter access patterns with fallbacks
These fixes address the most serious issues that would have prevented
the workflow from functioning correctly.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix critical wildcard mismatch: resource paths used {clusters} but rules didn't have this wildcard
- Get cluster count from config and substitute into resource paths
- Add missing cluster_network rule to generate clustered.nc input
- Fix all resource path references to use config-derived cluster values
- Ensure proper dependency chain: base → clustered → composed → solved
This resolves the fundamental mismatch between wildcard-based resource
paths and the new config-driven workflow approach.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
STANDARDIZATION IMPROVEMENTS: - Replace all direct config access with config_provider function calls - Ensure consistent use of resources() function from existing path provider - Follow PyPSA-EUR conventions for parameter handling in Snakemake rules - Use proper wildcard handling patterns from existing codebase KEY CHANGES: - get_compose_inputs: Use config_provider for all config value retrieval - cluster_network rule: Convert all params to use config_provider - compose_network rule: Standardize all parameter definitions - solve_network rule: Use config_provider for solver configuration - Collection rules: Use config_provider for run name retrieval - Validation rule: Standardize parameter access patterns This ensures the streamlined workflow follows the same patterns and conventions as the existing PyPSA-EUR workflow implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…y network creation
…ed workflow - Rename set_line_nom_max to cap_transmission_capacity with clearer parameter names - s_nom_max_set → line_max (AC line capacity limit) - p_nom_max_set → link_max (DC link capacity limit) - s_nom_max_ext → line_max_extension (AC line extension limit) - p_nom_max_ext → link_max_extension (DC link extension limit) - Port cap_transmission_capacity to compose_network.py for network composition - Port enforce_autarky to solve_network.py for pre-solve constraints - Add cap_transmission_capacity config section to config.default.yaml - Remove .get() calls in favor of strict dictionary access for better error messages - All functionality now config-based instead of wildcard-based 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add adjust_renewable_capacity_limits() function to compose_network.py for brownfield scenarios in myopic foresight mode - Function subtracts existing renewable capacities from previous horizons from p_nom_max values in current horizon - Uses renewable_carriers list from config (not hardcoded) - Remove redundant add_land_use_constraint() from solve_network.py for myopic mode (functionality now in compose step) - Keep add_land_use_constraint() for perfect foresight (different logic) - Add comprehensive test suite with 8 test cases covering: * Basic capacity subtraction * Multiple existing generators * Cases where existing exceeds potential * Negative value clipping * All renewable carriers * Edge cases (no existing, non-renewables) * Custom carrier lists - Update compose.smk to pass renewable_carriers as param 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- drop update_config_from_wildcards (deleted upstream) and the orphaned
scripts/temporal_aggregation.py
- set_temporal_aggregation now lives in prepare_sector_network and takes a
clustering.temporal-shaped dict, so cba.msv_extraction.resolution moves from
a '24H' string to averaging/segmentation/representative
- prepare_network lost its renewable_carriers argument
- read the {horizon} wildcard instead of {planning_horizons}
Accept "all" in the clustering schema again, restore the admin clustering CI target, and drop dead scenario blocks from test configs.
Bumps the github-actions group with 1 update: [dawidd6/action-download-artifact](https://github.com/dawidd6/action-download-artifact). Updates `dawidd6/action-download-artifact` from 21 to 23 - [Release notes](https://github.com/dawidd6/action-download-artifact/releases) - [Commits](dawidd6/action-download-artifact@v21...v23) --- updated-dependencies: - dependency-name: dawidd6/action-download-artifact dependency-version: '23' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- migrate config/examples to the new config format (drop scenario:, add planning_horizons/sector.enabled, restructure co2_budget) - drop dead wildcard kwargs and fix renamed rules in mock_snakemake - fix stale busshapes path, plotting rule ref and validation bot target
The function lost its only call site in the streamlined workflow; remove it along with the now-unused os and scipy.stats.beta imports.
Seven calls named rules that no longer exist (pre-existing on master) and retrieve_seawater_temperature wildcarded on horizon instead of year.
Repoint plot_base_statistics and development.smk at the new file names, gate sector plots on sector.enabled, drop stray heatmap output. Name CO2 global constraints via shared co2_limit_name() helper (CO2Limit-upper/-lower) and update summary and balance map consumers. Allow lower-only CO2 budgets.
Bump version to v2026.08.0, sort the accumulated release notes into Features / Changes / Bugfixes and Compatibility, and add PR references that were missing from individual entries. Also fix `[rule_name][]` cross-references, which have been broken across the whole documentation since PyPSA#2216 disabled the root ToC entry that carries the anchor, and document build_offshore_shapes and build_nuts3_shapes, which PyPSA#2169 added without a docs section. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Use scripts() helper throughout, restore dropped resources blocks and pdf summary plots, drop stale osm if-guard, require pandas offsets for clustering.temporal.averaging, validate temporal exclusivity in pydantic.
# Conflicts: # .gitignore # doc/release_notes.md
Brings in the review-comment round, the v2026.08.0 release, the cost-year override and the tidied benchmark/rule names. Fork-side follow-ups: `clustering.temporal.averaging` and `cba.msv_extraction.resolution.averaging` are now pandas offsets, `costs.custom_cost_fn` joins `custom_cost_scn`, and the stale osm if-guard and dead `build_carbon_budget` are dropped.
…/streamline-merge
Collapse repeated per-key ternaries in the compose_network input functions into single conditional blocks for h2_topology_tyndp, retro_endogen and enhanced_geothermal. DAG-equivalent for all test configs.
The grouped conditional blocks dropped input keys entirely, while prepare_sector_network reads h2_grid_tyndp, interzonal_prepped, retro_cost and floor_area unconditionally. A gated helper blanks the paths instead of removing the keys.
Author
|
Here is a more comprehensive investigation on the the model diff to open-tyndp master using the test config for tyndp and comparing 2040. In a nutshell, the difference is very small, only on the optimization outputs and all related to upstream changes that are on pypsa-eur master and not in open-tyndp yet, independent of streamline refactor. The report pins where the changes likely come from. I would expect them to vanish completely as soon as release Note The following content was written with AI and the product of multi iterations and reviews 2040 network diff:
|
| Diff | Cause / evidence | Introduced in | |
|---|---|---|---|
| D1 | CO₂ sequestration cost moved from store capex to link flow cost: link marginal_cost 30.011 vs 0.011 €/t, fixed at p_nom=inf; stores capital_cost 0 vs 30. Sequestration total unchanged (5.2953 Mt), vintage split moves (2040 e_nom_opt 5.17 vs 2.09 Mt). |
Commit drops capital_cost from the Store and adds marginal_cost to the Link. Master charged 30 €/t only on the one extendable vintage (62.8 M€ in the LP) while statistics.capex() reported 158.9 M€; the branch charges the full flow (158.9 M€, all in the objective). The ~96 M€ gap that closes is the objective delta. |
#2161 · 81bcb35 |
| D2 | Balkan liquid fuels: shipping oil BA00 1.4 vs 116.8, ME00 1.8 vs 19.9, MK00 0 vs 4.8 MW; kerosene RS00 263 vs 811 MW; AL00 land transport oil +42 %. Aggregate: kerosene −0.69 %, shipping oil −0.48 %. | fill_missing_years chained groupby(...).ffill().bfill(); the ffill returns a Series so the bfill ran ungrouped and data-less countries inherited the next country's totals. Affected set (AL, BA, ME, MK, RS) = exactly the non-EU group without JRC-IDEES coverage. |
#2243 · cf71d13 |
| D3 | onwind capital_cost +3.222 €/MW (55 gens, +0.04 %), same in all horizons. |
add_electricity_grid_connection now prefers the distribution grid reinforcement cost entry over electricity grid connection. |
#2161 · 81bcb35 |
| D4 | BA/ES shapes differ (ES 14 vs 13 polygons, area equal to 6 dp); 61 *-2035 p_nom values differ in the 5th–6th digit. |
Locks pin the same shapely 2.1.2 / geopandas 1.1.4; only pandas differs (3.0.3 vs 2.3.3). Environment artifact, cosmetic. | #2260 · f0812f2 |
D5–D7 — from the streamlined workflow (#1838)
All three are not affecting the test config as the transmission expansion it not enabled in the tyndp test cases. It only becomes relevant if that changes.
| Diff | Cause / evidence | Introduced in | |
|---|---|---|---|
| D5 | lv_limit per horizon: master 1.5548e8 MWkm in 2030/2035/2040; branch 1.5548 / 1.5548 / 1.8220e8, i.e. exactly Σ p_nom·length each year. Master's 2040 limit sat 14.7 % below existing capacity. |
Master's rule prepare_network has no {horizon} wildcard, so set_transmission_limit runs once on the 2030 grid; the TYNDP reference grid grows to 221 DC links / 410.8 GW by 2040. The branch calls prepare_network_for_solving inside per-horizon compose_network. |
051060d |
| D6 | All 221 DC links: p_nom_max = p_nom + 30000 MW (was + 20000). |
Master's limit_individual_line_extension applies lines.max_extension (20000) to lines and DC links; links.max_extension: 30000 was in the config but never read. cap_transmission_capacity reads s_nom_max_extension / p_nom_max_extension separately. |
0ca08c8, 341064a |
| D7 | DC capital_cost +8.1 % (214 links). Median €/km branch 5.18 / 4.90 / 4.70 per horizon vs master flat ≈ 4.35. Reported transmission capex 780 vs 721 M€. |
set_transmission_costs is byte-identical; only the costs frame differs. Default costs: year changed 2050 → unset, falling back to the horizon year. Master priced transmission at 2050 while pricing generators at the horizon. |
051904f |
lisazeyen
added
SB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this Pull Request
Integrates the streamline workflow proposed in PyPSA#1838 - please take a look there to get the whole context around this refactoring work. At this stage this effort serves two purposes. First, during this exercise we can test the completeness of the migration guide in the referenced pr. Secondly, it serves as a starting point for the proper merge proposal which we can spin of from here.
The major point of integration happens in the compose.smk where the inputs for the tyndp workflow is appended to the large inputs dict which goes into
compose_network. Whether this should live in a separate file I am happy to discuss and adjust to the needs.This is still in draft mode and I want to iterate with the CI a little and check with the group which workflows to focus on.
Tasks
We should likely merge the latest release of pypsa-eur first and then rebase this pr. This will resolve all the failing security check fails in the CI.
Workflow
Open issues
Notes
Checklist
Required:
doc/release_notes.md.If applicable:
scripts/lib/validation.config/test/*.yaml.pixi run -e open-tyndp tyndp-cyears-test).doc/*.mdfiles.doc/index.md.