Adjust EIA-930 timestamps

[rouille]

Purpose

Refactor notebook that calculates the correlation between CEMS timeseries and EIA-930 timeseries to find eventual shift in timestamps. The correlation is now calculated for a given year for each combination of BA, month and shift in hour from -12h to + 12h. The shift leading to the best correlation is reported in a table. To illustrate, the table here reports all BAs where the best shift is not always 0h, which is what we should observe once the current set of manual adjustments performed in the oge.eia930 module is applied.

In the table, only 4 BAs have a non 0h best shift in 2024: CISO, AVA, TAL and BANC.

BANC

EIA-930 looks to be shifted by 1h compared to CEMS around March 10th, 2024 and re-align early in April

TAL

The correlation analysis suggest a 1h shift in EIA-30 compared to CEMS for several months but the visual inspection does not clearly support this so I keft it unchanged. See June timeseries for example.

AVA

It looks like there is a 1h shift during DST (March 10th to November 3rd). The correlation analysis suggests that there is no shift in June, July and September but the visual inspection show some shift for most of the days in this month so I shifted the entire period. Note that it would make sense to shift all the month if it is coming from DST.
Start around March 11th

End around November 3rd

Month where correlation analysis suggest 0h shift

CISO

For CISO, I have shifted all the months from November 2023 (included) to December 2024 (excluded). The correlation analysis suggest a 6h shift in December that I cannot confirm visually (see plot below) so it left unchanged.

What the code is doing

It shifts timestamps.

Testing

Checks that new adjustments result in 0h best shift for month where changes have been introduced.

Where to look

The manual_930_adjust function in the oge.eia930 module

Usage Example/Visuals

N/A

Review estimate

10min

Future work

N/A

Checklist

• Update the documentation to reflect changes made in this PR
• Format all updated python files using black
• Clear outputs from all notebooks modified
• Add docstrings and type hints to any new functions created

[Copilot]

Pull request overview

This PR adjusts EIA-930 timestamp data based on correlation analysis with CEMS (Continuous Emissions Monitoring System) data. The analysis identified timestamp misalignments in several Balancing Authorities (BAs), and the PR implements corrections for these misalignments.

Key Changes

• Added timestamp adjustments for CISO (November 2023 - November 2024), AVA (March-November 2024 DST period), and BANC (March 10 - April 1, 2024)
• Refactored correlation analysis notebook to calculate correlations by month and year with improved handling of edge cases
• Added type hints and improved documentation across multiple functions in the eia930 module

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/oge/eia930.py	Added timestamp adjustments for CISO, AVA, and BANC; added type hints and improved docstrings for multiple functions
src/oge/column_checks.py	Added "energy_source_code_num" to dtype definitions
notebooks/manual_data/identify_eia930_time_lags.ipynb	Refactored correlation calculation to work monthly with improved data filtering and handling
notebooks/manual_data/update_utility_name_ba_map.ipynb	Updated year range from 2022-2023 to 2023-2024; removed unused imports
notebooks/manual_data/manually_identify_crosswalk_updates.ipynb	Updated year from 2023 to 2024; updated function call parameters
Pipfile.lock	Updated multiple dependencies to newer versions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rouille]

New file after adjustments made in this PR
cems_shifted_eia930_lags.csv

[grgmiller]

Thanks Ben. I also checked out this branch and took a look at CAISO in December 2024. I think the strange correlation offset (6h) is because while the CEMS profile continues to dip mid-day, there is some other, non-CEMS generators that are peaking mid-day, which obfuscates the dip in the 930 data and makes the mid-day period somewhat anti-correlated:

However, looking closely at this, I think there is still the same 1 hour offset present that we're seeing in the rest of 2024. Thus before merging, I'd recommend removing the end date of 12-01-2024 from the -1 shift you implemented for CAISO data (it is likely that this continues into 2025 until we assess otherwise).