perf: fix mainnet disk-IO saturation (transactions ordering index, mint-breakdown raw-table scan) - #91
Merged
Conversation
…tions ordering index get_mint_breakdown_between_dates summed reimbursement from the raw mod_to_acct_transfers table (~2.9 B rows on mainnet). The only index on op_reason is single-column, so every call heap-fetched all ~18 M reimbursement rows (~21 GB of disk reads, ~30 s) regardless of the date window. The same escrow amount is already recorded as a synthetic opReason 10 mint on event_claim_settleds, so the raw-table branch was redundant and double counted in the reward_distribution_detailed era. reimbursement is now derived from event_claim_settleds.mints only. Add transactions_block_id_desc_live_idx, a covering partial btree on transactions (block_id DESC, _id) INCLUDE (_block_range) WHERE id IS NOT NULL, so the PostGraphile transactions connection (ORDER BY block_id DESC LIMIT n + count(*)) no longer parallel seq scans the 13 GB heap twice per call. Created CONCURRENTLY and IF NOT EXISTS via the existing performance-index mechanism; it already exists on mainnet.
oten91
marked this pull request as ready for review
September 2, 2026 21:14
jorgecuesta
approved these changes
Sep 2, 2026
oten91
added a commit
that referenced
this pull request
Sep 3, 2026
…TE (#92) ## Summary Adds one performance index through the same mechanism as #91 (`getPerformanceIndexSqls` in `src/mappings/dbFunctions/domainRewards.ts`, executed by `createIndexesConcurrently`, `CONCURRENTLY` + `IF NOT EXISTS`, non-fatal on failure). Uses the `${dbSchema}` placeholder so beta gets it too. ```sql CREATE INDEX CONCURRENTLY IF NOT EXISTS balances_block_range_upper_idx ON <schema>.balances (upper(_block_range)); ``` ## Why Every block with balance changes runs this statement (historical mode): ```sql UPDATE "mainnet"."balances" SET "_block_range" = int8range(lower("_block_range"), $1, $2) WHERE upper("_block_range") = $3 ``` `balances` on mainnet is 53.8 M rows, 9.5 GB heap, 24 GB of indexes. The existing indexes are the SubQuery-generated GiST `(col, _block_range)` ones plus btrees on `id` / `_id` / `last_updated_block_id`. None can serve `upper(_block_range) = $3`, so the planner did a full index-only scan of a 2.7 GB GiST index on every call. Measured on the mainnet primary (`pg_stat_statements`): | | before | after | |---|---|---| | disk read per call | ~7.7 GB | 0 | | time per call | ~7.7 s | ~4 ms | | calls | 24,184 lifetime (~180 TB read), ~20/hour | 165 live calls overnight | The index was created ad hoc on the mainnet primary on 2026-09-02 22:41Z and verified; 372 MB, built in 25 s with `CONCURRENTLY`. `IF NOT EXISTS` makes the indexer start-up a no-op there. ## Where the statement comes from Not SubQuery core. It is our own code: `updateBalances` in `src/mappings/bank/balanceChange.ts`, step 2 ("reopen records that were closed for this block") issues `BalanceModel.model.update({ __block_range: int8range(lower(_block_range), null, '[)') }, { where: upper(_block_range) = blockId })` as a crash-recovery step before recomputing balances for the block. The index is the whole fix; the statement itself is unchanged. ## Not in this PR `get_mint_breakdown_between_dates_v2` and `idx_mod_to_acct_transfers_op_reason` are intentionally untouched. ## Verification - `tsc --noEmit`: no errors in `domainRewards.ts` (the pre-existing errors in `primitives.ts` / `query_client.ts` are unrelated: missing generated proto types and a cosmjs version mismatch in this checkout). - Index DDL is the exact statement already running on mainnet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a disk-IO saturation problem on the mainnet Postgres primary (
pg-ha-cluster-2). pnf-ops investigation on 2026-09-02: theexplorer-mainnetDB (1.7 TB) was reading 0.5–2 GB/s from disk around the clock. Two queries were responsible.Change 2 is also a correctness fix:
get_mint_breakdown_between_dateshas been reporting exactly 2× the real reimbursement amount for every block since #78. Verified on mainnet, see below.Change 1 — covering partial index on
mainnet.transactionsThe PostGraphile SQL behind
query transactionsList(transactions(first, offset, orderBy: BLOCK_ID_DESC) { totalCount nodes {...} }) parallel-seq-scanned the 13 GB heap twice per call: once forcount(*), once forORDER BY block_id DESC LIMIT n. That is ~22 GB of disk reads and 7.5 s per call, polled every ~30 s by external clients.Only GIST
(col, _block_range)indexes existed. GIST cannot serve ordering, and_block_range @> $1matches 100% of live rows so it filters nothing. PostGraphile addsid IS NOT NULLto the WHERE clause, which is why the index is partial on that predicate: it allows an index-only scan for the count.Measured on the primary after building it ad hoc: list half 25 buffers; count half index-only scan, zero disk reads, 2.4 s from cache. Size 2.1 GB, ~40 s build with
CONCURRENTLY.The index already exists on mainnet. This PR adds it to
getPerformanceIndexSqlsinsrc/mappings/dbFunctions/domainRewards.tsso a redeploy or--force-cleandoes not lose it. It goes through the existingcreateIndexesConcurrentlypath (one statement per query, no transaction, non-fatal on error) and isIF NOT EXISTS, so it is a no-op on mainnet.Change 2 —
get_mint_breakdown_between_datesno longer scansmod_to_acct_transfersThe first statement in the function was:
mod_to_acct_transfershas 2.86 B rows / 502 GB heap. The only index onop_reasonis single-column (idx_mod_to_acct_transfers_op_reason), so the planner walks all ~18 M reimbursement rows and heap-fetches each one, then hash-joins to the date-filtered blocks.EXPLAIN (ANALYZE, BUFFERS)for a one-day window: 2,795,359 blocks read from disk = 21 GB, ~30 s, regardless of window size. The function is called ~1.5×/min (money.pocket.network calls it 3× per render, plus external GraphQL clients), so this alone is ~500 MB/s of disk reads.The value it computes is dust on mainnet: over the last ~1000 blocks the reimbursement op_reason summed to 481,799 upokt (0.48 POKT) across 168,164 transfers, versus 268,872 POKT of supplier shareholder rewards in the same window.
What this PR does
Drops the raw-table branch.
reimbursementis now derived only fromevent_claim_settleds.mints(opReason10), which the second statement in the function already computed asreimbursement_v2_amount. That statement is unchanged; it usesidx_event_claim_settleds_block_idand is cheap.Why option (b) instead of reading from
mod_to_acct_transfers_summarizedThe rollup table was the preferred option, but two things argue against it here:
The raw branch was double counting in the current era.
_buildSettlementFromDetailedDistributioninsrc/mappings/pocket/relays.tsmaps everyreward_distribution_detailedentry intomod_to_acct_transfers, including theESCROW_DAO_TRANSFERrows, and also pushes a syntheticopReason 10mint holding their sum onto theevent_claim_settledsrow. The old SQL summed both and added them, so one chain transfer was reported twice for every block since Feature/0.1.33 events #78. Reading the same rows from the summarized table would carry the double count over.Verified on mainnet (pnf-ops, 2026-09-02), blocks > 904000:
mod_to_acct_transfers_summarized, op_reasonTLM_GLOBAL_MINT_REIMBURSEMENT_REQUEST_ESCROW_DAO_TRANSFERevent_claim_settleds.mints,opReason10Identical, so the old function returned exactly 2× the real value for that range.
The summarized table is not needed. It is fully backfilled on mainnet (min
block_id96845, same as the raw table), so option (a) would have worked, but it would only reproduce the same number themintspath already provides.Using the
mintsarray only means each era is counted once:mintscontent for reimbursementreward_distribution(oldest)opReason 10settlement_resultreward_distribution_detailed(#78, current)opReason 10The middle era is undercounted, but that was already the case for
reimbursement_v2_amount, and the amount involved is dust (see above). If exact history for that era matters, a targeted backfill ofmintsis the fix, not a 21 GB scan per call.Minor: the three returned fields are now wrapped in
COALESCE(..., 0). Before, an empty window returnednullforinflationandmint_burn(andnullforreimbursement, since0 + NULLisNULL).No partial index was added on
mod_to_acct_transfersfor thatop_reason; that option was considered and rejected (indexing a 502 GB table to speed up a no-op sum).Follow-up (not in this PR)
idx_mod_to_acct_transfers_op_reason(25 GB, 26,782 scans in its lifetime) exists only for this query and is what leads the planner into the 21 GB plan. Once this function is deployed and no longer uses it, it can be dropped in a follow-up. It is not dropped here.Verification
eslinton the two changed files: clean.tsc --noEmiterrors are pre-existing (missing codegen types, vendor) and none are indbFunctions.dbSchema = 't'and ran them on a throwawaypostgres:16container with fixtureblocks,event_claim_settleds, andtransactionstables. Function returns the expected sums ({"reimbursement": 10, "inflation": 7, "mint_burn": 150}for the in-window fixture; all zeros for an empty window). Index is created with the expected definition, and re-running the script is a no-op (already exists, skipping).Notes for the reviewer
get_mint_breakdown_between_dates_v2exists on the mainnet DB but not in this repository (no hit in any branch or commit); it was created by hand. Callers overwhelmingly hit v1 (26,450 calls vs 39 for_v2), so replacing v1 covers the load. A redeploy will not touch_v2; it should be dropped or redefined by hand.idx_transfers_recent_block_recipientis also not defined in this repository.