[indexer]: drain PendingStatusMetadata on every chain instance#929
Merged
Conversation
Add entry/exit logs in handlePendingStatusFlush, per-entityType fetch counts and parent-missing logs in PendingStatusService.flushBatch, and a modulo: 10 filter on the BlockHandler so it fires every 10 Hyperbridge blocks (~1 min) for easier log inspection.
If the entityType index is stale or stores values that drift from our
constants the per-entityType fetch returns zero rows even when the DB
has matching pending entries. Switch to PendingStatusMetadata.getByFields([], { limit })
which runs a single un-filtered query and dispatch on each row's
entityType field after read. Unknown entityType values are warn-logged
and skipped.
SubQuery multichain forces historical='timestamp', so the beforeFind hook attaches a __block_range @> <instance_timestamp> filter to every read. Rows written by another chain's instance are invisible to the Hyperbridge instance until Hyperbridge's indexed timestamp catches up to the writing chain's timestamp — which is why the Hyperbridge-only BlockHandler returned 0 rows despite the table having 21 entries. Move handlePendingStatusFlush out of the isHyperbridgeChain guard and add handlePendingStatusFlushEvm with kind: ethereum/BlockHandler so each chain instance flushes the rows it itself wrote (those rows are by construction in its own visible time range). Substrate modulo 10 (~1 min on 6s chains); EVM modulo 50 (chain-dependent, ~10 min on mainnet). Log lines include chainId so cross-chain runs are easy to distinguish.
The previous catch blocks used // @ts-ignore on the line above a multi-line logger.error call, so the directive didn't reach the error.message access on a later line and tsc failed with TS18046: 'error' is of type 'unknown'. Narrow with `error instanceof Error` instead of suppressing.
Empty getByFields filter is technically valid (x-sequelize 1992-1998 returns "" for Op.notIn:[]) but yields no diagnostic signal when 0 rows come back. Switch to an indexed-field IN filter — entityType is @index-ed — and log up to 3 sample rows (entityType@chain:id) so the chain that actually wrote stuck rows is identifiable from the indexer logs. Also lets us rule out empty-filter quirks definitively.
Add a temporary diagnostic block in flushBatch that probes one known-existing BSC pending row (commitment 0x10bf3434...c447) via three different store APIs: get(id), getByCommitment, and getByFields with '=' on entityType. Each takes a different @subql/node-core code path; whichever returns the row tells us where the bug is and which API to use as the fix. Also log block.timestamp on the EVM handler so we can rule out catch-up lag definitively. Will be reverted once the fix is identified.
Drop the diagnostic block + per-block entry/completed logs and per-row 'not yet present' info logs from the pending-status flush path. The original per-request flushPendingStatuses(commitment) is the working path; the new BlockHandlers stay as a no-op safety net (they log only when a real row is found or materialized). Also bump polkadot-hub-mainnet startBlock 16215883 → 16215884 and pin unfinalizedBlocks: false.
…flush # Conflicts: # sdk/packages/indexer/scripts/templates/substrate-chain.yaml.hbs # sdk/packages/indexer/src/handlers/events/pendingStatus/handlePendingStatusFlush.event.handler.ts # sdk/packages/indexer/src/services/pendingStatus.service.ts
ddboy19912
approved these changes
May 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
though the
PendingStatusMetadatatable had 21 entries. Root cause:SubQuery multichain forces
historical: 'timestamp'(
@subql/node-corestore.service.js:241-250) and thebeforeFindhook injects
WHERE __block_range @> <instance_timestamp>on everyread. Each chain's indexer runs as its own instance with its own
indexed timestamp, so rows written by chain B are invisible to chain A
until A's indexed timestamp catches up to B's write timestamp. That
also explains why the per-request
flushPendingStatuses(commitment)inside
createOrUpdatewas silently missing rows: same blind spot.handlePendingStatusFlushout of the{{#if isHyperbridgeChain}}guard in the substrate YAML template (
modulo: 10, ~1 min on a 6schain) so every substrate chain instance drains its own writes.
handlePendingStatusFlushEvmaskind: ethereum/BlockHandleronevery EVM chain (
modulo: 50) for the same reason — without it,rows written by EVM instances would only drain when their chain
happens to next touch the row by commitment, which never happens.
PendingStatusService.flushBatchtogetByFields([], { limit })with on-row
entityTypedispatch instead of a per-entityTypeindexed lookup, plus structured logs (
chain=,block=,fetched N,not yet present) for future post-mortem.