Skip to content

Add detailed EIA-176 continuation-line tables#5245

Draft
MeadBarrel wants to merge 15 commits into
catalyst-cooperative:mainfrom
MeadBarrel:i5240-300
Draft

Add detailed EIA-176 continuation-line tables#5245
MeadBarrel wants to merge 15 commits into
catalyst-cooperative:mainfrom
MeadBarrel:i5240-300

Conversation

@MeadBarrel

@MeadBarrel MeadBarrel commented May 15, 2026

Copy link
Copy Markdown
Contributor

Overview

Closes #5240.

What problem does this address?

EIA-176 continuation-line data includes useful detail behind several aggregate gas supply and disposition fields, but these records were only available in raw_eia176__continuation_text_lines. This PR exposes the detailed records for imports, supplemental gaseous fuels, exports, and other disposition as core tables while preserving the existing aggregate disposition table.

What did you change?

  • Added four EIA-176 core tables from continuation-line records:
    core_eia176__yearly_gas_imports, core_eia176__yearly_supplemental_gaseous_fuel_supplies,
    core_eia176__yearly_gas_exports, and core_eia176__yearly_gas_disposition_other.
  • Normalized continuation-line state/reference codes, fuel types, transportation modes, destination classifications, and lightly normalized free-text Line 18.4 disposition types.
  • Aggregated duplicate continuation-line records into stable table rows and added bounded total checks against the corresponding reported company-level values.
  • Added resource/field metadata, EIA-176 fuel-type enums, Alembic migrations, dbt source schemas, and row-count expectations for the new tables.
  • Updated core_eia176__yearly_gas_disposition metadata to point users to the new detailed exports and other-disposition tables.

Documentation

Make sure to update relevant aspects of the documentation:

  • Update the release notes: reference the PR and related issues.
  • Update relevant Data Source jinja templates (see docs/data_sources/templates).
  • Update relevant table or source description metadata (see src/metadata).
  • Review and update any other aspects of the documentation that might be affected by this PR.

Testing

Suggested reviewer checks:

pixi run --frozen pytest --no-cov test/unit/metadata_test.py::test_field_definitions test/unit/metadata_test.py::test_resource_descriptors_valid
pixi run --frozen dg check defs --verbose
pixi run dbt_helper validate --asset-select "key:core_eia176__yearly_gas_imports or key:core_eia176__yearly_supplemental_gaseous_fuel_supplies or key:core_eia176__yearly_gas_exports or key:core_eia176__yearly_gas_disposition_other"

To-do list

  • If updating analyses or data processing functions: make sure to update row count expectations in dbt tests.
  • Run pixi run prek-run to run linters and static code analysis checks.
  • Run pixi run pytest-ci locally to ensure that the merge queue will accept your PR.
  • Review the PR yourself and call out any questions or issues you have.
  • For PRs that change the PUDL outputs significantly, run the full ETL locally and then run the data validations using dbt. If you can't run the ETL locally then run the build-deploy-pudl GitHub Action manually and ensure that it succeeds.

@MeadBarrel MeadBarrel changed the title Add core_eia176__yearly_gas_imports (Eia176, line 3.0) Add detailed EIA-176 continuation-line tables May 19, 2026
@e-belfer e-belfer self-requested a review May 20, 2026 15:04
@e-belfer e-belfer added new-data Requests for integration of new data. eia176 Issues related to the EIA Form 176 natural gas supply and disposition dataset. labels May 20, 2026

@e-belfer e-belfer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're making great headway! Some requests for field changes and document updates, and a few questions.

Comment thread src/pudl/metadata/resources/eia176.py Outdated
Comment thread src/pudl/metadata/resources/eia176.py
Comment thread src/pudl/metadata/resources/eia176.py Outdated
Comment thread src/pudl/transform/eia176.py Outdated
Comment thread src/pudl/metadata/resources/eia176.py
Comment thread src/pudl/transform/eia176.py Outdated
Comment thread src/pudl/transform/eia176.py Outdated
Comment thread src/pudl/metadata/fields.py
Comment thread src/pudl/metadata/fields.py Outdated
Comment thread src/pudl/transform/eia176.py
@MeadBarrel

Copy link
Copy Markdown
Contributor Author

Ok, so i currently have reference_state renamed to supplier_location_code alongside supplier_location_type (subnational/national_or_other) for line 300, mimicking line 1400's destination_code and destination_type. Company name columns are named supplier_name and recipient_name.

I'm thinking, however, maybe it would be better to unify those, like counterparty_location_code, counterparty_location_type and counterparty_name?

@e-belfer

e-belfer commented Jun 9, 2026

Copy link
Copy Markdown
Member

@MeadBarrel thanks for all of your work on this one! I think it's ready to go, but because it's such a big pr I've tagged @cmgosnell for secondary review.

@cmgosnell Tagging you for secondary review, after alembic update and resolution of merge conflicts.

@e-belfer e-belfer requested a review from cmgosnell June 9, 2026 22:15
@cmgosnell cmgosnell added the community Issues that contributors have volunteered to take on or fostering more community label Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community Issues that contributors have volunteered to take on or fostering more community eia176 Issues related to the EIA Form 176 natural gas supply and disposition dataset. new-data Requests for integration of new data.

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

Create detailed tables from raw_eia176__continuation_text_lines

3 participants