Integrate EIA860 2025 early release data by aesharpe · Pull Request #5324 · catalyst-cooperative/pudl

aesharpe · 2026-06-12T20:07:32Z

Overview

Closes #5322.

What problem does this address?

Adds EIA860 early release data for 2025.

What did you change?

Update DOI
Update working paritions
Update package data mapping
Update alembic migration
Update full/fast etl settings file
Update release notes
Update codes with new values/fixes
Util/plant mapping
Write function to drop plants/utils with all NA rows

To Do:

Row count updates

Documentation

Make sure to update relevant aspects of the documentation:

Update the release notes: reference the PR and related issues.
Update relevant Data Source jinja templates (see docs/data_sources/templates).
Update relevant table or source description metadata (see src/metadata).
Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

If updating analyses or data processing functions: make sure to update row count expectations in dbt tests.
Run pixi run prek-run to run linters and static code analysis checks.
Run pixi run pytest-ci locally to ensure that the merge queue will accept your PR.
Review the PR yourself and call out any questions or issues you have.
For PRs that change the PUDL outputs significantly, run the full ETL locally and then run the data validations using dbt. If you can't run the ETL locally then run the build-deploy-pudl GitHub Action manually and ensure that it succeeds.

krivard

Early review: this looks good! ty for adding the null annotations; it helps to know where all our little edge cases came from

aesharpe · 2026-06-17T15:44:53Z

How do I get rid of this annoying typo error? https://results.pre-commit.ci/run/github/80646423/1781710886._grFgw6ATLCnQUCeqcFcJA I don't understand why it is triggered for this...

krivard · 2026-06-17T15:54:06Z

There's a typos guide at: https://docs.catalyst.coop/pudl/en/latest/dev/dev_setup.html#linting-and-formatting

aesharpe · 2026-06-19T17:30:20Z

 core_eia860__assn_boiler_generator,2024,11659
+core_eia860__assn_boiler_generator,2025,9244


Seems like a big jump down, but I'm not sure what to do about it.

aesharpe · 2026-06-19T17:31:01Z

-core_eia860__scd_generators,2025,36743
+core_eia860__scd_generators,2025,36172


Weird that this went down?

Looks like most of the old values that aren't in the update are retired:

makes sense!

aesharpe · 2026-06-19T17:31:28Z

-core_eia860__scd_plants,2025,17183
+core_eia860__scd_plants,2025,16980


Same here, why did this go down?

All the rows dropped have all NA rows isna().all():

aesharpe · 2026-06-19T17:32:26Z

-out_eia923__monthly_generation_fuel_by_generator,2004,204260
-out_eia923__monthly_generation_fuel_by_generator,2005,204252
-out_eia923__monthly_generation_fuel_by_generator,2006,205404
-out_eia923__monthly_generation_fuel_by_generator,2007,210791
-out_eia923__monthly_generation_fuel_by_generator,2008,214087
-out_eia923__monthly_generation_fuel_by_generator,2009,216918
-out_eia923__monthly_generation_fuel_by_generator,2010,220660
-out_eia923__monthly_generation_fuel_by_generator,2011,225723
-out_eia923__monthly_generation_fuel_by_generator,2012,232886
-out_eia923__monthly_generation_fuel_by_generator,2013,236797
-out_eia923__monthly_generation_fuel_by_generator,2014,241035
-out_eia923__monthly_generation_fuel_by_generator,2015,244904
-out_eia923__monthly_generation_fuel_by_generator,2016,253095
-out_eia923__monthly_generation_fuel_by_generator,2017,260396
-out_eia923__monthly_generation_fuel_by_generator,2018,269534
-out_eia923__monthly_generation_fuel_by_generator,2019,277348
-out_eia923__monthly_generation_fuel_by_generator,2020,284582
-out_eia923__monthly_generation_fuel_by_generator,2021,299112
-out_eia923__monthly_generation_fuel_by_generator,2022,308204
+out_eia923__monthly_generation_fuel_by_generator,2004,204272
+out_eia923__monthly_generation_fuel_by_generator,2005,204264
+out_eia923__monthly_generation_fuel_by_generator,2006,205416
+out_eia923__monthly_generation_fuel_by_generator,2007,210803
+out_eia923__monthly_generation_fuel_by_generator,2008,214099
+out_eia923__monthly_generation_fuel_by_generator,2009,216942
+out_eia923__monthly_generation_fuel_by_generator,2010,220684
+out_eia923__monthly_generation_fuel_by_generator,2011,225747
+out_eia923__monthly_generation_fuel_by_generator,2012,232910
+out_eia923__monthly_generation_fuel_by_generator,2013,236821


What's with the big change here?

There were three generators at three different plans that got added in retroactively. Not sure why? This is the change between the years of 2004 and 2024:

Assume its the same for the _by_energy_source table too

nice catch! are they newly-reporting plants submitting a bunch of backdata or is there some new mapping thing where maybe we used to drop them and now we don't?

It looks like those generators weren't in the raw data from before this DOI so I'm guessing it's some sort of backfilling on behalf of EIA?

krivard

all the non-rowcounts changes look good (the one suggestion is optional) and you seem to have the hunt for the weird rowcounts effects under control -- let me know when you've narrowed it down / want extra eyes on those surprise backdata additions

krivard · 2026-06-19T17:57:17Z

-core_eia860__scd_generators,2025,36743
+core_eia860__scd_generators,2025,36172


makes sense!

krivard · 2026-06-19T17:58:54Z

-core_eia860__scd_plants,2025,17183
+core_eia860__scd_plants,2025,16980


krivard · 2026-06-19T18:07:15Z

-out_eia923__monthly_generation_fuel_by_generator,2004,204260
-out_eia923__monthly_generation_fuel_by_generator,2005,204252
-out_eia923__monthly_generation_fuel_by_generator,2006,205404
-out_eia923__monthly_generation_fuel_by_generator,2007,210791
-out_eia923__monthly_generation_fuel_by_generator,2008,214087
-out_eia923__monthly_generation_fuel_by_generator,2009,216918
-out_eia923__monthly_generation_fuel_by_generator,2010,220660
-out_eia923__monthly_generation_fuel_by_generator,2011,225723
-out_eia923__monthly_generation_fuel_by_generator,2012,232886
-out_eia923__monthly_generation_fuel_by_generator,2013,236797
-out_eia923__monthly_generation_fuel_by_generator,2014,241035
-out_eia923__monthly_generation_fuel_by_generator,2015,244904
-out_eia923__monthly_generation_fuel_by_generator,2016,253095
-out_eia923__monthly_generation_fuel_by_generator,2017,260396
-out_eia923__monthly_generation_fuel_by_generator,2018,269534
-out_eia923__monthly_generation_fuel_by_generator,2019,277348
-out_eia923__monthly_generation_fuel_by_generator,2020,284582
-out_eia923__monthly_generation_fuel_by_generator,2021,299112
-out_eia923__monthly_generation_fuel_by_generator,2022,308204
+out_eia923__monthly_generation_fuel_by_generator,2004,204272
+out_eia923__monthly_generation_fuel_by_generator,2005,204264
+out_eia923__monthly_generation_fuel_by_generator,2006,205416
+out_eia923__monthly_generation_fuel_by_generator,2007,210803
+out_eia923__monthly_generation_fuel_by_generator,2008,214099
+out_eia923__monthly_generation_fuel_by_generator,2009,216942
+out_eia923__monthly_generation_fuel_by_generator,2010,220684
+out_eia923__monthly_generation_fuel_by_generator,2011,225747
+out_eia923__monthly_generation_fuel_by_generator,2012,232910
+out_eia923__monthly_generation_fuel_by_generator,2013,236821


nice catch! are they newly-reporting plants submitting a bunch of backdata or is there some new mapping thing where maybe we used to drop them and now we don't?

krivard · 2026-06-19T18:08:21Z

            (69312, "ECHEB"),
+            (37538, "PV1"),
+            (59525, "W8785"),
+            (67351, "WIL2B"),  # spellchecker:ignore


wild. spellchecker wyed

krivard · 2026-06-19T18:09:29Z

+17862,,,67469,DGI Rattler LLC,EIA860 ER 2025
+17863,,,67518,"Seat Solar Holdings, LLC",EIA860 ER 2025
+17864,,,67521,HWS Energy Holdings LLC,EIA860 ER 2025
+17865,,,67541,Delorean Power LLC,EIA860 ER 2025


Co-authored-by: Kathryn Mazaitis <1158666+krivard@users.noreply.github.com>

Update package data for eia860 2025 early release data

fabec17

github-project-automation Bot added this to Catalyst Megaproject Jun 12, 2026

github-project-automation Bot moved this to New in Catalyst Megaproject Jun 12, 2026

aesharpe self-assigned this Jun 12, 2026

aesharpe added eia860 Anything having to do with EIA Form 860 data-update When fresh data is integrated into PUDL from quarterly or annual updates labels Jun 12, 2026

aesharpe moved this from New to In progress in Catalyst Megaproject Jun 12, 2026

aesharpe added 2 commits June 12, 2026 16:08

Update release notes

54b1ca2

Merge branch 'main' into eia860-er-2026

18de7f7

aesharpe changed the title ~~Update package data for eia860 2025 early release data~~ Integrate EIA860 2025 early release data Jun 12, 2026

aesharpe added 7 commits June 12, 2026 16:16

Update migration

653dab4

Merge branch 'main' into eia860-er-2026

8a09e62

Update migration around main

d0d8a0d

Update lock file

3f533e3

Fix eia860_scd_storage table

1b9ac7b

Update emissions control codes

1e81da1

Fix boiler status code types

b4ef16a

krivard reviewed Jun 16, 2026

View reviewed changes

aesharpe added 6 commits June 16, 2026 09:51

Merge branch 'main' into eia860-er-2026

518b792

Map plants and remove one bad plant

4b77d66

Make NA row remover function more general and apply to utility NAs

397c7f3

Remove NA value util from util mapping spreadsheet

de742b8

Merge branch 'main' into eia860-er-2026

8d383e6

Fix util NA drop rows

82a339a

aesharpe added 3 commits June 17, 2026 10:56

Add spellcheck ignore line

003462f

Merge branch 'main' into eia860-er-2026

fba5589

Add row counts

3f8bfaf

aesharpe commented Jun 19, 2026

View reviewed changes

krivard reviewed Jun 19, 2026

View reviewed changes

aesharpe and others added 3 commits June 19, 2026 12:55

Update validation tests

c0f14c0

Update pixi.lock

a84e24a

make error message more specific

abdc4f6

Co-authored-by: Kathryn Mazaitis <1158666+krivard@users.noreply.github.com>

		core_eia860__assn_boiler_generator,2024,11659
		core_eia860__assn_boiler_generator,2025,9244

		core_eia860__scd_generators,2025,36743
		core_eia860__scd_generators,2025,36172

		core_eia860__scd_plants,2025,17183
		core_eia860__scd_plants,2025,16980

Uh oh!

Conversation

aesharpe commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What problem does this address?

What did you change?

Documentation

Testing

To-do list

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

aesharpe commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krivard commented Jun 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aesharpe commented Jun 12, 2026 •

edited

Loading

aesharpe commented Jun 17, 2026 •

edited

Loading

aesharpe Jun 19, 2026 •

edited

Loading

aesharpe Jun 19, 2026 •

edited

Loading

aesharpe Jun 19, 2026 •

edited

Loading