settler, aggregate-prover: resume unsettled bundles after a restart - #148
Draft
biryukovmaxim wants to merge 18 commits into
Draft
biryukovmaxim wants to merge 18 commits into
biryukovmaxim wants to merge 18 commits into
Conversation
…nant tip Config gains journal: Option<Arc<dyn SettlementJournal>>, wired only in the proving node; exec and test construction sites stay None, enforced by a spawn assert requiring the queue and the watch. The worker records each published bundle's geometry and deletes entries a settlement watch advance covers. Review riders from the journal crate: drop the unused vprogs-core-types dependency, assert that recording at the same start replaces the prior entry, and align the corrupted-key expect message.
…ce, not only at startup
Run 1 quiesces the pipeline (every straggler settlement landed, every in-flight bundle submitted) before mining the tail carriers, so no batch commits past the journal tail across the kill: such a gap leaves the warm store's state chain past the last settlement with no journal entry covering it, and the resumed prover then skips every bundle forever (recorded as a production follow-up). Bundle minimum 1 keeps a lone real carrier batch from stranding below the bundle minimum when a carrier block comes out empty. The restarted settler supersedes the re-fed tail without a second submission and settles new work past the landed tip.
A kill between a batch's commit and its bundle's journal record wedges the covenant: the committed range above the journal tail has no entry, the scheduler never re-schedules committed batches, so every resumed bundle proves from a state root the covenant never took and the settler skips it forever (the restarted-prover-idle signature). The startup resume now follows a committed-gap pass: with tail_end the pre-pass journal snapshot's max end and committed_tip the highest committed checkpoint, the range (tail_end, committed_tip] is re-formed from persisted batch metadata and cached per-batch receipts via the split_straddler recipe, recorded as one journal entry, and fed after the pending tail. A settlement boundary inside the range splits it like the straddled entry; empty batches (lane tip carried unchanged) prove no receipt and compose nothing, matching the live bundle filter. The journal gate no longer skips the gap check when the journal is empty: the whole committed span is then the gap, gated on the bridge's first tip publication with an escape on the first scheduled batch so a settlement-free covenant is never parked. committed_tip reaches the worker as a reverse BatchMetadata seek on the settlement journal (new trait method alongside the full-metadata read the gap pass needs for block hashes, seq commits, and lane tips).
…d gap Run 1 lands settlements, mines tail carriers under a bundle minimum of 4, and kills at a polled deterministic point: the carriers' batch metadata and per-batch receipts are durable while the sub-minimum park keeps their bundle from journaling (a leftover prefix that does reach the minimum is detected as journal growth and the round retries). Run 2 must re-form the committed range, settle it, and settle new work past it; without the startup gap pass it wedges with the pinned signature (bundles proved, every submission skipped, chain stuck), which the pre-fix tree was re-verified to reproduce. The spawn helper splits so a test can hand the node a caller-opened store handle and keep a clone for live column-family reads (journal, batch metadata, receipt presence) without contending the RocksDB lock. open_store_retrying's final resume_unwind now names the store path and the caught failure: resume_unwind bypasses the restored panic hook and the original panic ran under the silent hook, so a real lock timeout previously failed the test with no message at all.
Stale-boundary healing: the advance-pass scope extends to this run's gap entry (its end exceeds the pre-restart snapshot), so a settlement boundary the gap pass could not yet observe (the empty-journal escape preceding a catch-up burst's watch publication, or the non-empty branch's pre-downtime baseline) re-splits that entry in process instead of wedging until the next restart; the unmapped-boundary case keeps the re-feed-unchanged residual, now worded as such. split_straddler gains the emptiness predicate the gap pass established: empty batches prove no receipt and compose nothing, so a straddled suffix spanning one is skipped instead of dropping the entry uncovered. The gap bundle now covers the contiguously-durable prefix of the range: a receipt still in flight for a freshly committed batch bounds the bundle instead of aborting the whole pass, the entry's end fields derive from the last covered batch's own metadata, and only the range above a miss logs at error and stays uncovered (an in-flight batch journals live; a genuinely lost one remains the loud documented ceiling). Docs: the empty-journal wait names its park behavior (no tip and no batch parks until activity, matching pre-fix behavior); committed_tip's density premise is stated as a rollback ceiling rather than a guarantee.
… blind The confirm-wait warn tick's probe now distinguishes a spent covenant outpoint from a node drop: our own continuation UTXO live on chain confirms the settlement from the submission's own data (txid, block DAA score) without the settlement watch, a spent outpoint without it supersedes the bundle, and any probe error leaves the wait running exactly as before. The settler can no longer wait forever on a settlement the chain already holds. The mempool read also accepts the wRPC subsystem's plain not-found message alongside the typed variant: over wRPC the node's not-found arrives only as that message, so the old drop probe never fell through to its chain read against a real node.
One INFO line per settlement observed while processing a chain block (txid + block + daa), so a future settlement miss is visible at ingestion immediately: a fetch-stalled bridge shows VCC retry warnings with no settlement-block lines, while an extraction miss would show the block processed with no settlement line.
Reproduces the production wedge deterministically: the settler runs with a settlement watch nobody writes (the bridge-frozen incident shape), the settlement mines and accepts on the simnet L1 anyway, and the settler must confirm through the warn tick's chain probe, keep the covenant a single chain, and settle subsequent bundles. Without the backstop the single-flight settler parks forever at chain length 1.
The non-empty-journal startup gate waited for the bridge's first tip publication with no inbox escape, unlike the empty-journal branch. On a covenant whose first-ever settlement never landed (a fresh lane, or a TN5-shaped eviction striking bundle one) the bridge has no last settlement to republish, so the gate parked the whole worker until shutdown: the tail never re-fed and proving was held. Mirror the empty branch's inbox escape: the gate now breaks with no tip on the first scheduled batch and resume_pending takes the no-tip path, re-feeding the scoped tail unchanged (the sibling of the unmapped-boundary path). A settlement racing the escape reaches the main loop's advance pass, which re-runs the resume with the real tip.
Review nits and a comment-convention sweep over the branch diff: - The committed-gap miss log now says "covering nothing" when the miss is the first gap index, instead of claiming coverage through an index the loop never reached. - The resume split's suffix drops its unused checkpoint-index tuple slot; the receipts vector is built directly. - covenant_spent's doc states what it returns (spent, not spendable) and names the error case consistently. - Comment wording: a journal-free run (not journal), and the test's wallet-sizing rationale no longer references the debugging session that observed it.
The warm-restart drive loops assign settled on every iteration and the four adapted sites reassign it unconditionally before the next read, so the 0 initializer never survives to a read; unused_assignments rejects it under -D warnings. The two sites that read settled straight out of the loop keep the initializer, as the loop may not run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The incidents
Two tn10 production wedges with the same signature: the settler submits a settlement, the tx is mined and accepted within ~1 s, but the settlement watch never advances and the single-flight settler waits forever ("unconfirmed after Ns"). Nothing settles afterward while execution and proving continue; the node looks healthy from the L2 side. After a prover restart the lane stayed idle (TN5). Root cause (investigated on the live systems): the bridge's Full-verbosity
get_virtual_chain_from_block_v2catch-up fetch over a dust-storm window exceeded the RPC client's 60 s request timeout and entered a retry livelock, freezing the bridge tip below the settlement's accepting block; the notification-only confirm wait turned that transport stall into a permanent settler wedge. The same 60 s livelock is the long-known slow-link catch-up defect; the transport fix (chunked/reduced-verbosity catch-up) is separate and NOT part of this branch.What this branch does
Settlements resume after a prover restart, receipts are reused rather than re-proved (only the cheap aggregate re-runs), a competitor that settles during downtime is handled without forking, and the settler can no longer wedge on a mined-but-unobserved settlement.
state/settlement-journal, new): proved-but-unsettled bundles are journaled at artifact publication (end_index,from_block,block_prove_to,seq_commit) over a newStateSpace::SettlementJournalcolumn family; entries compact away against the covenant tip.Tests
two_provers_contendgrows from 7 to 13 integration tests (dev-mode proofs, simnet L1): warm restart settles the pending tail; competitor sweep during downtime; split at the competitor boundary with no new carriers; own settlement landing during downtime (no fork, no double submission, settles past the landed tip); committed-but-unjournaled gap coverage (red-verified against the pinned wedge); blind-watch confirm via the chain poll.Out of scope (open)
Related
read_batch_receipt)