Detect stale latest closed_at in no_missing_ledgers - #256
Open
amishas157 wants to merge 7 commits into
Open
Conversation
* contract token updates * changes for asset code * update codes * updates based on pr comments * Lag batch_end_date by 1 hour in eho_by_ops singular test (#254) * updates based on pr comments * Fix stale comment about contract_id fallback * fix joins in account balances contract model * add test --------- Co-authored-by: chowbao <simon.chow@stellar.org>
* accept batch start/end date and update test tag * add: no_negative_circulating_supply singular test tagged validate_history_tables Ports the circulating-supply check from datafold-monitors (anomaly-detection/no_negative_circulating_supply.yaml.j2) into a dbt singular test. References stg_token_transfers_raw, sums mint vs burn events per contract, and fails when any non-allowlisted contract has negative cumulative supply. Tagged validate_history_tables so the new airflow gate selects it via `dbt test --select tag:validate_history_tables`. * add: singular tests for state, EHO, and token_transfer validation gates Three new singular tests, each tagged with the matching airflow gate: - state_tables_missing_ledger_sequence (validate_state_tables): detects missed export batches by flagging unexpectedly large gaps (>240 ledgers, ~2x the 10-min export cadence) between consecutive distinct ledger_sequence values across 11 state staging tables. - eho_row_count (validate_eho_tables): asserts strict count equality between enriched_history_operations and stg_history_operations within the batch window. Fast complement to eho_by_ops which finds the individual missing op_ids. - token_transfers_no_negative_circulating_supply (validate_token_transfer_tables): mirrors the existing no_negative_circulating_supply test but reads from the token_transfers mart, catching any enrichment-introduced drift between raw events and the mart. * refactor: parameterize ledger_sequence_increment singular test Accept optional table_group_name and table_name vars so the check can target all tables, a named group, or a single table. Retag as validate_no_missing_ledgers. * add: populate table groups for ledger_sequence_increment test Fill history_tables, state_tables, snapshot_tables, current_tables, and fee_stats_tables so the parameterized test can be targeted by group. * remove: state_tables_missing_ledger_sequence singular test Superseded by the parameterized ledger_sequence_increment test, which can be targeted at the state_tables group. * remove stg_history_contract_events * remove stg_history_effects model * Move test to new file * fix braces * no pricing check with ledger sequence * add test for no missing days in snapshot * add new test * snapshot run at day level * Apply suggestion from @amishas157 * remove extranneous tests
The gap check only finds missing ledgers between existing rows, so a table that stops receiving data silently passes. Add a stale_latest check that fails when max(closed_at) is null or below batch_end_date, and tag failures by reason in the test output.
amishas157
commented
May 14, 2026
| {% endif %} | ||
| ) | ||
|
|
||
| , stale_latest as ( |
Contributor
Author
There was a problem hiding this comment.
This test ensures that data until batch_end_date(with 5 minutes lag threshold) is available in the table.
The downstream task checks for upstream tables.
chowbao
approved these changes
May 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
no_missing_ledgerssingular test only finds gaps between existing rows, so a table that stops receiving data silently passes.stale_latestCTE that fails whenmax(closed_at)isnull(no rows in the 7-day lookback) or< batch_end_date(latest data hasn't caught up).failuresCTE with afailure_reasoncolumn ('gap'or'stale_latest') so the test output makes the cause obvious. Severity stayserrorfor both.Test plan
batch_end_dateset ahead ofmax(closed_at)in one of the listed tables — expect astale_latestrow for that table.failure_reason = 'gap').table_group_name/table_namevar scoping still narrows both checks.