Skip to content

perf: fix mainnet disk-IO saturation (transactions ordering index, mint-breakdown raw-table scan) - #91

Merged
oten91 merged 1 commit into
mainfrom
fix/mainnet-disk-io-tx-index-mint-breakdown
Sep 2, 2026
Merged

oten91 merged 1 commit into
mainfrom
fix/mainnet-disk-io-tx-index-mint-breakdown

Conversation

@oten91

@oten91 oten91 commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes a disk-IO saturation problem on the mainnet Postgres primary (pg-ha-cluster-2). pnf-ops investigation on 2026-09-02: the explorer-mainnet DB (1.7 TB) was reading 0.5–2 GB/s from disk around the clock. Two queries were responsible.

Change 2 is also a correctness fix: get_mint_breakdown_between_dates has been reporting exactly 2× the real reimbursement amount for every block since #78. Verified on mainnet, see below.

Change 1 — covering partial index on mainnet.transactions

The PostGraphile SQL behind query transactionsList (transactions(first, offset, orderBy: BLOCK_ID_DESC) { totalCount nodes {...} }) parallel-seq-scanned the 13 GB heap twice per call: once for count(*), once for ORDER BY block_id DESC LIMIT n. That is ~22 GB of disk reads and 7.5 s per call, polled every ~30 s by external clients.

Only GIST (col, _block_range) indexes existed. GIST cannot serve ordering, and _block_range @> $1 matches 100% of live rows so it filters nothing. PostGraphile adds id IS NOT NULL to the WHERE clause, which is why the index is partial on that predicate: it allows an index-only scan for the count.

CREATE INDEX CONCURRENTLY IF NOT EXISTS transactions_block_id_desc_live_idx
  ON mainnet.transactions (block_id DESC, _id) INCLUDE (_block_range)
  WHERE id IS NOT NULL;

Measured on the primary after building it ad hoc: list half 25 buffers; count half index-only scan, zero disk reads, 2.4 s from cache. Size 2.1 GB, ~40 s build with CONCURRENTLY.

The index already exists on mainnet. This PR adds it to getPerformanceIndexSqls in src/mappings/dbFunctions/domainRewards.ts so a redeploy or --force-clean does not lose it. It goes through the existing createIndexesConcurrently path (one statement per query, no transaction, non-fatal on error) and is IF NOT EXISTS, so it is a no-op on mainnet.

Change 2 — get_mint_breakdown_between_dates no longer scans mod_to_acct_transfers

The first statement in the function was:

SELECT COALESCE(SUM(t.amount), 0) FROM mainnet.mod_to_acct_transfers t
JOIN mainnet.blocks b ON t.block_id = b.id
WHERE t.op_reason = 'TLM_GLOBAL_MINT_REIMBURSEMENT_REQUEST_ESCROW_DAO_TRANSFER'
  AND b.timestamp BETWEEN start_date AND end_date;

mod_to_acct_transfers has 2.86 B rows / 502 GB heap. The only index on op_reason is single-column (idx_mod_to_acct_transfers_op_reason), so the planner walks all ~18 M reimbursement rows and heap-fetches each one, then hash-joins to the date-filtered blocks. EXPLAIN (ANALYZE, BUFFERS) for a one-day window: 2,795,359 blocks read from disk = 21 GB, ~30 s, regardless of window size. The function is called ~1.5×/min (money.pocket.network calls it 3× per render, plus external GraphQL clients), so this alone is ~500 MB/s of disk reads.

The value it computes is dust on mainnet: over the last ~1000 blocks the reimbursement op_reason summed to 481,799 upokt (0.48 POKT) across 168,164 transfers, versus 268,872 POKT of supplier shareholder rewards in the same window.

What this PR does

Drops the raw-table branch. reimbursement is now derived only from event_claim_settleds.mints (opReason 10), which the second statement in the function already computed as reimbursement_v2_amount. That statement is unchanged; it uses idx_event_claim_settleds_block_id and is cheap.

Why option (b) instead of reading from mod_to_acct_transfers_summarized

The rollup table was the preferred option, but two things argue against it here:

  1. The raw branch was double counting in the current era. _buildSettlementFromDetailedDistribution in src/mappings/pocket/relays.ts maps every reward_distribution_detailed entry into mod_to_acct_transfers, including the ESCROW_DAO_TRANSFER rows, and also pushes a synthetic opReason 10 mint holding their sum onto the event_claim_settleds row. The old SQL summed both and added them, so one chain transfer was reported twice for every block since Feature/0.1.33 events #78. Reading the same rows from the summarized table would carry the double count over.

    Verified on mainnet (pnf-ops, 2026-09-02), blocks > 904000:

    source amount (upokt) entries
    mod_to_acct_transfers_summarized, op_reason TLM_GLOBAL_MINT_REIMBURSEMENT_REQUEST_ESCROW_DAO_TRANSFER 499,298 174,621
    event_claim_settleds.mints, opReason 10 499,298 174,621

    Identical, so the old function returned exactly 2× the real value for that range.

  2. The summarized table is not needed. It is fully backfilled on mainnet (min block_id 96845, same as the raw table), so option (a) would have worked, but it would only reproduce the same number the mints path already provides.

Using the mints array only means each era is counted once:

era mints content for reimbursement counted
reward_distribution (oldest) synthetic opReason 10 1×
settlement_result chain mints, no reimbursement entry 0×
reward_distribution_detailed (#78, current) synthetic opReason 10 1×

The middle era is undercounted, but that was already the case for reimbursement_v2_amount, and the amount involved is dust (see above). If exact history for that era matters, a targeted backfill of mints is the fix, not a 21 GB scan per call.

Minor: the three returned fields are now wrapped in COALESCE(..., 0). Before, an empty window returned null for inflation and mint_burn (and null for reimbursement, since 0 + NULL is NULL).

No partial index was added on mod_to_acct_transfers for that op_reason; that option was considered and rejected (indexing a 502 GB table to speed up a no-op sum).

Follow-up (not in this PR)

idx_mod_to_acct_transfers_op_reason (25 GB, 26,782 scans in its lifetime) exists only for this query and is what leads the planner into the 21 GB plan. Once this function is deployed and no longer uses it, it can be dropped in a follow-up. It is not dropped here.

Verification

  • eslint on the two changed files: clean. tsc --noEmit errors are pre-existing (missing codegen types, vendor) and none are in dbFunctions.
  • Rendered both SQL statements with dbSchema = 't' and ran them on a throwaway postgres:16 container with fixture blocks, event_claim_settleds, and transactions tables. Function returns the expected sums ({"reimbursement": 10, "inflation": 7, "mint_burn": 150} for the in-window fixture; all zeros for an empty window). Index is created with the expected definition, and re-running the script is a no-op (already exists, skipping).
  • Not verified: production plan/timing after deploy. The ad hoc index numbers above are from the primary; the function change has not been timed on mainnet.

Notes for the reviewer

  • get_mint_breakdown_between_dates_v2 exists on the mainnet DB but not in this repository (no hit in any branch or commit); it was created by hand. Callers overwhelmingly hit v1 (26,450 calls vs 39 for _v2), so replacing v1 covers the load. A redeploy will not touch _v2; it should be dropped or redefined by hand.
  • idx_transfers_recent_block_recipient is also not defined in this repository.

…tions ordering index

get_mint_breakdown_between_dates summed reimbursement from the raw
mod_to_acct_transfers table (~2.9 B rows on mainnet). The only index on
op_reason is single-column, so every call heap-fetched all ~18 M
reimbursement rows (~21 GB of disk reads, ~30 s) regardless of the date
window. The same escrow amount is already recorded as a synthetic
opReason 10 mint on event_claim_settleds, so the raw-table branch was
redundant and double counted in the reward_distribution_detailed era.
reimbursement is now derived from event_claim_settleds.mints only.

Add transactions_block_id_desc_live_idx, a covering partial btree on
transactions (block_id DESC, _id) INCLUDE (_block_range) WHERE id IS NOT
NULL, so the PostGraphile transactions connection (ORDER BY block_id
DESC LIMIT n + count(*)) no longer parallel seq scans the 13 GB heap
twice per call. Created CONCURRENTLY and IF NOT EXISTS via the existing
performance-index mechanism; it already exists on mainnet.
@oten91
oten91 marked this pull request as ready for review September 2, 2026 21:14
@oten91
oten91 requested a review from jorgecuesta September 2, 2026 21:14
@oten91
oten91 merged commit 720711d into main Sep 2, 2026
5 checks passed
oten91 added a commit that referenced this pull request Sep 3, 2026
…TE (#92)

## Summary

Adds one performance index through the same mechanism as #91
(`getPerformanceIndexSqls` in
`src/mappings/dbFunctions/domainRewards.ts`, executed by
`createIndexesConcurrently`, `CONCURRENTLY` + `IF NOT EXISTS`, non-fatal
on failure). Uses the `${dbSchema}` placeholder so beta gets it too.

```sql
CREATE INDEX CONCURRENTLY IF NOT EXISTS balances_block_range_upper_idx
  ON <schema>.balances (upper(_block_range));
```

## Why

Every block with balance changes runs this statement (historical mode):

```sql
UPDATE "mainnet"."balances"
SET "_block_range" = int8range(lower("_block_range"), $1, $2)
WHERE upper("_block_range") = $3
```

`balances` on mainnet is 53.8 M rows, 9.5 GB heap, 24 GB of indexes. The
existing indexes are the SubQuery-generated GiST `(col, _block_range)`
ones plus btrees on `id` / `_id` / `last_updated_block_id`. None can
serve `upper(_block_range) = $3`, so the planner did a full index-only
scan of a 2.7 GB GiST index on every call.

Measured on the mainnet primary (`pg_stat_statements`):

| | before | after |
|---|---|---|
| disk read per call | ~7.7 GB | 0 |
| time per call | ~7.7 s | ~4 ms |
| calls | 24,184 lifetime (~180 TB read), ~20/hour | 165 live calls
overnight |

The index was created ad hoc on the mainnet primary on 2026-09-02 22:41Z
and verified; 372 MB, built in 25 s with `CONCURRENTLY`. `IF NOT EXISTS`
makes the indexer start-up a no-op there.

## Where the statement comes from

Not SubQuery core. It is our own code: `updateBalances` in
`src/mappings/bank/balanceChange.ts`, step 2 ("reopen records that were
closed for this block") issues `BalanceModel.model.update({
__block_range: int8range(lower(_block_range), null, '[)') }, { where:
upper(_block_range) = blockId })` as a crash-recovery step before
recomputing balances for the block. The index is the whole fix; the
statement itself is unchanged.

## Not in this PR

`get_mint_breakdown_between_dates_v2` and
`idx_mod_to_acct_transfers_op_reason` are intentionally untouched.

## Verification

- `tsc --noEmit`: no errors in `domainRewards.ts` (the pre-existing
errors in `primitives.ts` / `query_client.ts` are unrelated: missing
generated proto types and a cosmjs version mismatch in this checkout).
- Index DDL is the exact statement already running on mainnet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants