Support country-specific mappings and custom transmission line types

[danielelerede-oet]

Description

This PR adds support for country-specific base voltages and transmission line-type mappings while preserving the existing PyPSA-Earth mappings as the default.

The use of country-specific line types is configurable through lines.use_country_specific_types.

When this option is enabled, country-specific mappings are used only if mappings are available for every configured country. Otherwise, the complete default mapping is used. Availability is evaluated separately for AC and DC mappings, so country-specific and default mappings are not mixed within the same category.

The country associated with a line is determined from bus0, consistently across base-network construction, network simplification, and augmented connections. The closest available voltage is selected from the chosen mapping.

Configuration

Line-type mappings remain in the configuration under:

lines:
  use_country_specific_types: false

  ac_types:
    default:
      # voltage-to-line-type mappings
    AU:
      # country-specific mappings
    US:
      # country-specific mappings

  dc_types:
    default:
      # voltage-to-line-type mappings

The currently existing mapping is stored under default. Optional country-specific mappings can be added using country codes.

• If lines.use_country_specific_types is false, the complete default mapping is used.
• If it is true and mappings exist for every configured country, the country-specific mappings are used.
• If at least one configured country is missing from a category, the complete default mapping is used for that category.
• AC and DC mapping availability is evaluated independently.
• The closest available voltage in the selected mapping is used.
• The country of each line is determined from the country of bus0.

For example, if every configured country has an AC mapping but the DC mappings are incomplete, country-specific mappings are used for AC lines while the complete default mapping is used for DC lines.

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented, including updates to docstrings for meaningful functions.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/user-guide/configuration.md and doc/tutorials/electricity-model.md.
• If config sections were added, renamed, or removed, update doc/assets/scripts/extract_config_snippets.py accordingly.
• Archives of the uploaded data do not have an enclosing folder and archive names correspond to the conventions of configs/bundle_config.yaml.
• A note for the release notes doc/release-notes.md is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

Many thanks @danielelerede-oet !
Really like the proposal that enables custom line-types by country that is a long wanted feature! Adding a comment

[davide-f]

Could we keep the assignment of line types in the config file and add the csv for the characteristics of custom line types that are not defined in pypsa?

The advantage is that for each country we can define only the relevant line types and avoid nan values. The higher the number of countries, the more complex the handling becomes with csv format

[danielelerede-oet]

The point is that this should address a slightly different case. In this PR, the csv file doesn't define the electrical characteristics of custom line types. It is supposed to only map each country and voltage level to a line type which is already existing in PyPSA (where applicable, as in the case of Australia).

The idea of having custom line types assigned to countries as the US where the mapping with already existing PyPSA line types is not accurated would be complementary to this country-specific assignment mechanism rather than a replacement for it.

Keeping the assignment in the config would also preserve the current limitation, since the purpose of this PR is precisely to support different mappings by country without adding country-specific configuration sections. Let me know what you think :)

[danielelerede-oet]

Hi @davide-f @ekatef I have extended the PR beyond the mapping you checked in initial review that only accounted for countries with line types already defined in PyPSA (i.e. Australia).

The PR now also covers the US case, where representative types are not available in the standard library. I added a central data/custom_line_types.csv registry containing US electrical parameters derived from the ACTIVSg82k synthetic network, together with a non-workflow script that makes the derivation reproducible.

base_network.py now loads these additional definitions into n.line_types before assigning types and calculating line capacities. I also updated simplify_network.py and augmented_line_connections.py so country-specific types are retained when voltage layers are consolidated or new HVAC connections are introduced.

The existing default mapping remains unchanged for countries without dedicated mappings.

[davide-f]

Amazing Daniele! I'm adding few comments; this is a great feature!

As a comment: have you verified that linetypes properly propagate across scripts especially after cluster_network, just to avoid surprises?
When linetype is defined, pypsa internally may recalculate values by using linetype info and it is advisable to ensure the information is preserved.

[danielelerede-oet]

Amazing Daniele! I'm adding few comments; this is a great feature!

As a comment: have you verified that linetypes properly propagate across scripts especially after cluster_network, just to avoid surprises? When linetype is defined, pypsa internally may recalculate values by using linetype info and it is advisable to ensure the information is preserved.

Yes @davide-f this should be ensured but I'm testing the model on the US to double-check

[danielelerede-oet]

Amazing Daniele! I'm adding few comments; this is a great feature!

As a comment: have you verified that linetypes properly propagate across scripts especially after cluster_network, just to avoid surprises? When linetype is defined, pypsa internally may recalculate values by using linetype info and it is advisable to ensure the information is preserved.

@davide-f, this comment identified an important point. I checked the networks generated across the different workflow steps and found that the voltage-specific US line types are correctly assigned in elec.nc, but simplify_network subsequently remaps every line to the type associated with the configured base voltage. With the current 380 kV base voltage, the US mapping has no corresponding entry and therefore falls back to the default Al/St 240/40 4-bundle 380.0 type.

The procedure preserves s_nom by recalculating num_parallel, and the dependent electrical parameters are populated before solving. However, the original US-specific impedance characteristics are not retained directly, since r, x, and b are recalculated from the fallback 380 kV type.

We could either retain the current behaviour or revise the simplification settings to use more representative regional base voltages and mapped line types: potentially US_345kV at 345 kV for the US, and the currently mapped Al/St 240/40 4-bundle 380.0 type at a 330 kV base voltage for Australia.

A preliminary comparison on the same simplified network confirms that using a 345 kV base voltage with US_345kV:

• preserves s_nom exactly
• increases num_parallel by 41.7%
• decreases r by 28.6%
• increases x by 4.4%
• increases b by 77.6%.

Therefore, the regional simplification mapping is not merely a naming change: it materially affects the electrical representation, even though the thermal capacities remain unchanged. I will implement the US-specific mapping and rerun the network to quantify its impact on flows, congestion, dispatch and the objective.

I'm now testing this in pypsa-meets-earth/pypsa-northamerica#43 : electricity.base_voltage is now defined as a single mapping in config.default.yaml:

base_voltage:
  default: 380.
  AU: 330.
  US: 345.

In simplify_network_to_base_voltage(), the simplification voltage is resolved from the country of each bus, falling back to default when no country-specific value is configured. Lines are consequently assigned the corresponding country-specific voltage and line type, while num_parallel is recalculated to preserve s_nom, as before.

For the current US-only test, all buses and lines are simplified at 345 kV and use US_345kV as mentioned above.

One crucial point remains open before using this implementation in multi-country networks: an AC interconnector could connect countries to which different simplification voltages are assigned. The current logic follows the country of bus0, consistently with the existing line-type selection, but this could leave a regular Line connecting buses with different nominal voltages.

I have not added a fail-fast test because we should first decide whether interconnected countries should share a common regional base voltage or whether voltage transitions should be represented explicitly (y preference would be to use a common simplification voltage for countries belonging to the same interconnected network).

Result of test in PyPSA-NorthAmerica (PR #43)

The US_345kV line type definition is now preserved through both simplification and spatial clustering, all 219 clustered lines referencing it have a valid associated linetype and the optimization completes successfully with slight changes to the objective function (+0.3%). This confirms that the country-specific voltage and line-type mapping is applied correctly without changing the lines’ thermal capacities.

[ekatef]

@danielelerede-oet @davide-f looks an absolutely amazing work!

@danielelerede-oet that is a very impactful improvement you have implemented to preserve the line type definition.

We have experienced dropping of custom line parameters also in PyPSA-Zambia model, and can confirm that it's essential to get the linetype right. However, the fix we have applied has been an ad hoc solution. Tagging @tacwebservices who has worked a lot on examining regional representation the power grid and its' effects on modelling.

[davide-f]

Thanks @danielelerede-oet ! Added a minor change.
As this PR adds a breaking change in the use of line_types, it is advisable to signal backcompatibility.
Yerbol has integrated an option to track migrations:

pypsa-earth/scripts/_helpers.py

Line 117 in 2970624

CONFIG_MIGRATIONS = [

Please add the config references there to support the migrations. From a functional perspective I have only a comment left.
Many thanks!

[davide-f]

I'd revise the implementation: when country-specific options exist, the defaults are not used entirely, otherwise we risk mixing default and non-default linetypes for countries for small voltage differences

[danielelerede-oet]

Yeah, it's addressed now. It was already in my mind. Now the default and country-specific mappings are no longer merged. Country-specific line types are only used when lines.use_country_specific_types (as suggested by @ekatef ) is enabled and mappings are available for every configured country. Otherwise, the complete default mapping is used. This is evaluated separately for AC and DC line types, so default and country-specific line types are never mixed within the same category.

[danielelerede-oet]

Thanks @danielelerede-oet ! Added a minor change. As this PR adds a breaking change in the use of line_types, it is advisable to signal backcompatibility. Yerbol has integrated an option to track migrations:

pypsa-earth/scripts/_helpers.py

Line 117 in 2970624

CONFIG_MIGRATIONS = [

Please add the config references there to support the migrations. From a functional perspective I have only a comment left. Many thanks!

@davide-f I checked CONFIG_MIGRATIONS, but I don’t think this change belongs there because lines.ac_types and lines.dc_types keep the same paths, while only the internal structure changes.

Since CONFIG_MIGRATIONS introduced by Yerbol only handles path renames, I added a dedicated handler in _helpers.py to detect legacy top-level voltage mappings, moves them under default, preserves country-specific mappings, and emits a FutureWarning.

[danielelerede-oet]

Thanks @davide-f I applied your latest comments too

[ekatef]

@carlosfv92 this PR may resolve the issues with the transmission capacity you reported in #1820