Skip to content

pdp: use the reorg-check hash index for the pending message queue panel - #1481

Open
TippyFlitsUK wants to merge 1 commit into
filecoin-project:mainfrom
TippyFlitsUK:fix/message-queue-panel-hash-index
Open

TippyFlitsUK wants to merge 1 commit into
filecoin-project:mainfrom
TippyFlitsUK:fix/message-queue-panel-hash-index

Conversation

@TippyFlitsUK

Copy link
Copy Markdown
Collaborator

Priority

Not urgent. GUI-only cost; nothing functional is affected.

The problem

The pending eth messages query in web/api/webrpc/message_queue.go joins message_sends_eth on the computed key lower(trim(both from mse.signed_hash)). No plain-column index can serve that, so every render of the message queue panel hash-joins against a full scan of message_sends_eth:

->  Hash Right Join
      Hash Cond: (lower(TRIM(BOTH FROM mse.signed_hash)) = mwe.signed_tx_hash)
      ->  Seq Scan on message_sends_eth mse  (rows=5687145)

On my two calibnet SPs (5.5M and 5.7M send rows) that is 8.5 s per render, logged over 1,300 times in one node's recent log window. Nodes with smaller send tables or an empty pending queue stay under the 5 s slow-log threshold, which is why it only surfaces on high-volume nodes.

The matching index already exists: 20260527-pdpv0-reorg-chk-indexes.sql ships idx_message_sends_eth_signed_hash_norm on exactly this expression, partial on send_success = true AND signed_hash IS NOT NULL. The panel query cannot use it because a bare LEFT JOIN must also consider rows outside that predicate.

The change

Rewrite the join as a LATERAL probe with send_success = TRUE, which makes the existing partial index legal and turns the full scan into one point probe per pending message:

->  Index Scan using idx_message_sends_eth_signed_hash_norm on message_sends_eth mse
      Index Cond: (lower(TRIM(BOTH FROM signed_hash)) = mwe.signed_tx_hash)
Execution Time: 64.693 ms

8,459 ms to 64.7 ms on the same node and data. No schema change. Sends with send_success = false are excluded from the join; a wait row is only inserted after successful submission, so no pending wait matches such a send.

Verification

Running on both calibnet SPs. Zero slow-SQL warnings since restart; the panel returns identical results.

The pending eth messages query joins message_sends_eth on the computed
key lower(trim(both from signed_hash)). No plain-column index can serve
that, so every render of the message queue panel hash-joins against a
full scan of message_sends_eth: 8.5 s per render on calibnet SPs with
5.5M+ send rows. Nodes with smaller send tables or an empty pending
queue stay under the 5 s slow-log threshold.

The matching index already ships: 20260527-pdpv0-reorg-chk-indexes.sql
creates idx_message_sends_eth_signed_hash_norm on exactly this
expression, partial on send_success = true AND signed_hash IS NOT NULL.
A bare LEFT JOIN cannot use it because it must also consider rows
outside that predicate.

Rewrite the join as a LATERAL probe with send_success = TRUE, making the
partial index legal and turning the full scan into one point probe per
pending message: 8,459 ms to 64.7 ms on the same node and data, no
schema change. Sends with send_success = false are excluded from the
join; a wait row is only inserted after successful submission, so no
pending wait matches such a send.
@TippyFlitsUK
TippyFlitsUK requested review from a team and ZenGround0 as code owners August 30, 2026 16:36
@FilOzzy FilOzzy added the team/fs-wg Items being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10 label Aug 30, 2026
@FilOzzy FilOzzy added this to FOC Aug 30, 2026
@github-project-automation github-project-automation Bot moved this to 📌 Triage in FOC Aug 30, 2026
@TippyFlitsUK TippyFlitsUK self-assigned this Aug 30, 2026
@FilOzzy FilOzzy moved this from 📌 Triage to 🔎 Awaiting review in FOC Aug 30, 2026

@LexLuthr LexLuthr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why we were doing the LEFT join in the first place? It just doesn't make sense to do a JOIN here.

@github-project-automation github-project-automation Bot moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Aug 31, 2026
@TippyFlitsUK
TippyFlitsUK force-pushed the fix/message-queue-panel-hash-index branch from 8a09a37 to 0f456e1 Compare August 31, 2026 18:39
@TippyFlitsUK

Copy link
Copy Markdown
Collaborator Author

It came in with the panel itself in #1351 - the join is only there to show send_reason and send_time next to each pending hash. message_waits_eth only holds the hash and status, so the two tables have to be combined somewhere to render the panel. Happy to restructure it however you'd prefer - or drop the reason/time columns if that's the better call.

@TippyFlitsUK
TippyFlitsUK requested a review from LexLuthr August 31, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team/fs-wg Items being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants