Skip to content

Release v20260804 - #309

Open
taha-aziz wants to merge 7 commits into
masterfrom
release-v20260804
Open

Release v20260804#309
taha-aziz wants to merge 7 commits into
masterfrom
release-v20260804

Conversation

@taha-aziz

@taha-aziz taha-aziz commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Release v20260804 for stellar-dbt-public.

What's in this release

Merging auto-tags the next release; stellar-dbt's packages.yml gets repinned from the release-v20260804 branch to that tag before the stellar-dbt release PR merges.

amishas157 and others added 7 commits July 30, 2026 11:01
@amishas157
Tune incremental strategies (merge → table / insert_overwrite / micro…
9b42861 
…batch) (#302)

* Tune incremental strategies for non-aggregated models

- current_state (10 models): pin incremental_strategy='merge' explicitly
  (was relying on BigQuery's implicit default). Correct SCD-1 upsert grain.
- history_assets: incremental(merge) -> table. Tiny cumulative dimension;
  full rebuild removes incremental-state/orphan risk at negligible cost.
  asset_id test switched from windowed incremental_unique to plain unique.
- evicted_keys: incremental(merge) -> table. Unpartitioned, so the hourly
  merge already full-scanned the table each run; a full rebuild is
  comparable cost and removes cleanup risk. Added whole-table
  unique_combination_of_columns on [ledger_key_hash, closed_at].

* table for daily_fee_stats_agg and trade_agg aggregates

Both are day-grained aggregates that were on implicit merge. Full rebuild
removes incremental-state/cleanup risk. trade_agg keeps its month
partition_by (valid + useful for pruning on a table). unique_key dropped
(unused by table).

* Remove incremental behavior from full-rebuild table models (#304)

* Remove incremental behavior from full-rebuild table models

daily_fee_stats_agg, evicted_keys, and trade_agg are materialized as table
but still carried is_incremental() blocks, batch_start_date filters, and a live
batch_end_date upper bound. In a partial_backfill the batch_end_date is a past
date, so the full rebuild would truncate to the window end and the whole-table
swap would publish a truncated table over live (silent data loss).

Strip all batch-date coupling so these tables always rebuild whole history and
are immune to window truncation under any run mode.

* Make microbatch batch_size overridable via DBT_MICROBATCH_BATCH_SIZE env var

Each microbatch model hardcoded batch_size = 'year' if flags.FULL_REFRESH
else 'day'. Read an optional DBT_MICROBATCH_BATCH_SIZE env var instead, falling
back to that same default when it is unset/empty:

  {% set batch_size = env_var('DBT_MICROBATCH_BATCH_SIZE', '')
       or ('year' if flags.FULL_REFRESH else 'day') %}

This lets a run (e.g. a large partial_backfill) coarsen the batch granularity
to month/year without a code change. env_var (not a dbt var / --vars) is used so
it works under the backfill build's in_pod_retries wrapping, which is
incompatible with --vars. Runs that do not set the env var behave exactly as
before.

* Convert incremental test variants to standard tests on table models

Models converted to a table materialization were still using the incremental
test variants (stellar_dbt_public.incremental_not_null / incremental_unique /
incremental_unique_combination_of_columns), which only assert over the recent
window -- a remnant of their incremental past. A full-rebuild table should be
tested whole, so:

- incremental_not_null -> not_null (condition -> config.where)
- incremental_unique -> unique
- incremental_unique_combination_of_columns -> dbt_utils.unique_combination_of_columns
- add a uniqueness test where the model had none after dropping unique_key
  (int_asset_stats_agg, tvl_agg_recognized_assets, tvl_agg_usd)

Only table-materialized models are touched; incremental/microbatch/merge models
keep their incremental test variants.

* Refactor int_trade_agg_* to produce complete history on full rebuild

trade_agg is now materialized as a table, but its four int_trade_agg_*
parents only computed the batch_start_date reference day, so a rebuild
dropped all prior trade days. Refactor the parents to emit complete
history keyed on the real trade date:

- int_trade_agg_day: day_agg is now cast(ledger_closed_at as date) and the
  single-day window filter is removed, so every historical day is emitted.
- int_trade_agg_week/month/year: rewritten to build a per-(day, pair) base
  and compute the trailing 7/30/365-day rolling metrics with window
  functions (range over unix_date(day_agg)), producing a row for every day
  a pair traded instead of one reference day per run. Rolling open = first
  trade in the window, close = latest day's close, matching prior semantics.

Grain of all four is (day_agg, asset_a, asset_b). Note: unlike the old
per-run snapshot, dormant pairs are no longer carried forward on days they
did not trade; rows exist only for (day a pair traded, pair).

* Revert trade_agg to incremental and drop int_trade_agg_* refactor

Keep trade_agg materialized as incremental (merge, unique_key
[day_agg, asset_a, asset_b]) with its incremental tests, and restore the
four int_trade_agg_* parents to their original batch-window builds. This
undoes the table conversion and the full-history parent refactor; the
merge strategy continues to retain prior trade days across runs.

* update sequence of columns

* Always enable concurrent_batches and copy_partitions on microbatch models

Set concurrent_batches=true and copy_partitions=true unconditionally on all
microbatch-strategy models (previously gated on flags.FULL_REFRESH, so they
only took effect during full refreshes). The batch_size full-refresh sizing
is left unchanged.

* history_assets: drop dead is_incremental() block

It's a table model, so is_incremental() is always false and only the
full-refresh branch ever ran. Remove the whole conditional and keep the
unconditional full dedup + rebuild.

* Centralize microbatch batch_size in a macro

All 14 microbatch models carried a byte-identical copy of

    env_var('DBT_MICROBATCH_BATCH_SIZE', '') or ('year' if flags.FULL_REFRESH else 'day')

so the resolution policy could only be changed by editing every model. Replace
it with microbatch_batch_size(), which documents the reasoning in one place:

- DBT_MICROBATCH_BATCH_SIZE is how Airflow overrides the granularity for a
  single run, mostly for partial backfills. It is an env var rather than
  --vars because the backfill build wraps dbt in a bash retry loop that is not
  compatible with --vars.
- year on a full refresh, because rebuilding all history in day-sized batches
  is thousands of BigQuery jobs.
- day otherwise, matching the scheduled cadence.

Behavior is unchanged: an empty env var still falls through to the flag, and a
set one still wins. Verified with dbt parse that all 14 models resolve to day
by default, to month under DBT_MICROBATCH_BATCH_SIZE=month, and that the env
var takes precedence over the full-refresh branch.

The macro takes default/full_refresh_default arguments so a model partitioned
coarser than day can pass its own, since bq_validate_microbatch_config requires
batch_size to be at least as coarse as partition_by.granularity. All 14 models
are day-partitioned today and use the defaults.

The macro lives here rather than in stellar-dbt so this repo stays buildable on
its own CI; stellar-dbt reaches it as stellar_dbt_public.microbatch_batch_size().

The flags.FULL_REFRESH branch is carried over as-is. dbt discourages flags as
parse-time config input because a cached partial parse can go stale; the macro
documents why that is accepted and how to avoid it locally.

* Trim microbatch_batch_size doc comment to the resolution rules

The header had grown to 35 lines of commentary for a 10-line macro. Keeps the
three resolution rules, the granularity constraint that the arguments exist for,
and usage. The dbt parse-time-flags caveat is dropped from the file; it is
recorded in 97915ed's commit message.
@hunterpack @claude
sources: env-driven database/schema; microbatch_begin() for CI windows (
765eebe 
#308)

* sources: env-driven database/schema; microbatch_begin() for CI windows

Replaces the need for the parent project's deprecated source override
(stellar-dbt models/sources/dev_sources.yml, 'overrides:'), which resolves
nondeterministically on CI runners under dbt 1.12 and intermittently pointed
CI builds at prod (stellar/stellar-dbt#1067). Sources now carry the same
env-var resolution directly, so every environment resolves identically and
the override becomes deletable.

Also ports microbatch_begin() from the parent project and converts the 11
microbatch models' hardcoded begin dates to it, preserving each model's
original date as the non-ci default. Under target=ci the models walk a
30-day window instead of full history.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert the three kwarg-form begins missed in the first pass

int_account_balances__{trustlines,offers,liquidity_pools} set
begin='2021-01-01' as a config kwarg rather than the meta_config dict
form, so the first conversion missed them (Copilot review catch). All 14
public microbatch models now use microbatch_begin().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@hunterpack @claude
int_reflector_prices__sdex: warn severity for asset not_null tests ou…
f30c83c 
…tside prod

The model left-joins reflector feeds to stg_assets; assets outside the CI
sample window null the joined columns, so these tests hard-fail CI on
sample coverage rather than code. Same conditional-severity pattern as the
snapshot ymls: error in prod, warn elsewhere. First surfaced by the
full-graph slim-ci run on stellar-dbt#1069 once CI builds started
processing real rows (stellar-dbt#1067 era fixes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hunterpack @claude
account_balances intermediates: warn severity for contract_id not_nul…
7db980b 
…l outside prod

Same sample-coverage class as the reflector tests: these models process
real rows in CI for the first time (microbatch_begin windows, #308) and
classic entries legitimately lack contract mappings inside a 2-day sample
window. House conditional-severity pattern: error in prod, warn elsewhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hunterpack @claude
sdex snapshot: warn severity for asset not_null tests outside prod
6bf9e41 
Same two sample-coverage null rows as int_reflector_prices__sdex, one
level downstream in the snapshot. Same house pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hunterpack @claude
daily_agg marts: warn severity for contract_id not_null outside prod
8f886c5 
Same lineage nulls as the account_balances intermediates, one level down.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hunterpack @claude
account_balances__daily_agg: warn severity for the day/account/contra…
c10985c 
…ct uniqueness test outside prod

Null contract_ids from sample coverage collapse rows onto the same key;
in prod the column is populated and the key is unique. Local full-graph
verification (no fail-fast) confirms this is the last remaining failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 31, 2026 21:38
@taha-aziz
taha-aziz requested a review from a team as a code owner July 31, 2026 21:38
Copilot AI reviewed Jul 31, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prepares release v20260804 by improving dbt incremental processing and making source resolution deterministic across environments.

Changes:

  • Adds environment-driven source database/schema configuration.
  • Tunes table, merge, and microbatch materializations with configurable CI windows.
  • Adjusts data tests for full-table models and non-production environments.

Reviewed changes

Copilot reviewed 58 out of 58 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
models/sources/src_ttl.yml Adds environment-driven source location.
models/sources/src_trust_lines.yml Adds environment-driven source location.
models/sources/src_token_transfers_raw.yml Adds environment-driven source location.
models/sources/src_restored_key.yml Adds environment-driven source location.
models/sources/src_offers.yml Adds environment-driven source location.
models/sources/src_liquidity_pools.yml Adds environment-driven source location.
models/sources/src_history_transactions.yml Adds environment-driven source location.
models/sources/src_history_trades.yml Adds environment-driven source location.
models/sources/src_history_operations.yml Adds environment-driven source location.
models/sources/src_history_ledgers.yml Adds environment-driven source location.
models/sources/src_history_effects.yml Adds environment-driven source location.
models/sources/src_history_contract_events.yml Adds environment-driven source location.
models/sources/src_history_assets.yml Adds environment-driven source location.
models/sources/src_contract_data.yml Adds environment-driven source location.
models/sources/src_contract_code.yml Adds environment-driven source location.
models/sources/src_config_settings.yml Adds environment-driven source location.
models/sources/src_claimable_balances.yml Adds environment-driven source location.
models/sources/src_accounts.yml Adds environment-driven source location.
models/sources/src_accounts_signers.yml Adds environment-driven source location.
models/snapshots/reflector_prices_data_sdex_snapshot.yml Downgrades selected non-production failures to warnings.
models/marts/tokens/transfers/token_transfers.sql Applies shared microbatch settings.
models/marts/ledger_fee_stats_agg.sql Applies shared microbatch settings.
models/marts/ledger_current_state/ttl_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/trust_lines_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/offers_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/liquidity_pools_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/contract_data_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/contract_code_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/config_settings_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/claimable_balances_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/accounts_current.sql Makes merge strategy explicit.
models/marts/ledger_current_state/account_signers_current.sql Makes merge strategy explicit.
models/marts/hourly_soroban_fee_agg_contract.sql Applies shared microbatch settings.
models/marts/hourly_fee_agg_account.sql Applies shared microbatch settings.
models/marts/history_assets.yml Replaces incremental tests with whole-table tests.
models/marts/history_assets.sql Converts asset history to a table rebuild.
models/marts/evicted_keys.yml Adds whole-table integrity tests.
models/marts/evicted_keys.sql Converts evicted keys to a table rebuild.
models/marts/enriched_history/enriched_history_operations.sql Applies shared microbatch settings.
models/marts/enriched_history/enriched_history_operations_soroban.sql Applies shared microbatch settings.
models/marts/daily_fee_stats_agg.yml Replaces incremental tests with whole-table tests.
models/marts/daily_fee_stats_agg.sql Converts daily fee statistics to a table rebuild.
models/marts/asset_balances/asset_balances__daily_agg.yml Adjusts non-production test severity.
models/marts/asset_balances/asset_balances__daily_agg.sql Applies shared microbatch settings.
models/marts/account_balances/account_balances__daily_agg.yml Adjusts non-production test severity.
models/marts/account_balances/account_balances__daily_agg.sql Applies shared microbatch settings.
models/intermediate/tvl/int_tvl_trustlines.sql Applies shared microbatch settings.
models/intermediate/tvl/int_tvl_accounts.sql Applies shared microbatch settings.
models/intermediate/tokens/transfers/int_token_transfer_enrichment.sql Applies shared microbatch settings.
models/intermediate/reflector_prices/int_reflector_prices__sdex.yml Adjusts non-production test severity.
models/intermediate/account_balances/int_account_balances__trustlines.sql Applies shared microbatch settings.
models/intermediate/account_balances/int_account_balances__offers.yml Adjusts non-production test severity.
models/intermediate/account_balances/int_account_balances__offers.sql Applies shared microbatch settings.
models/intermediate/account_balances/int_account_balances__liquidity_pools.yml Adjusts non-production test severity.
models/intermediate/account_balances/int_account_balances__liquidity_pools.sql Applies shared microbatch settings.
models/intermediate/account_balances/int_account_balances__contracts.yml Adjusts non-production test severity.
macros/microbatch_begin.sql Adds configurable CI microbatch windows.
macros/microbatch_batch_size.sql Adds configurable microbatch sizing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread models/marts/history_assets.sql
Comment on lines +23 to +27
, row_number() over (
partition by asset_id
order by batch_run_date asc
) as dedup_grouping
from {{ ref('stg_history_assets') }}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@taha-aziz @amishas157 @hunterpack