Skip to content

op-interop-filter: verdict RPC can block behind ingester write lock, turning slow ingestion into fail-closed timeouts #21943

Description

@claude

Found via Slack thread CI-flake investigation.

Problem

The verdict path (CheckAccessListValidateAccessEntry) reads ingester state via Contains and LatestTimestamp, which take c.mu.RLock (op-interop-filter/filter/logsdb_chain_ingester.go:243-253, :290-309). Meanwhile processBlockLogs holds c.mu.Lock across the entire logsDB block write — every AddLog plus the SealBlock (logsdb_chain_ingester.go:824-869, lock taken at :827).

So when ingestion stalls mid-write (slow disk, starved goroutine), verdict RPCs don't return a fast soft "not yet available" answer — they block on the lock. op-reth's verdict query is a one-shot with a hardcoded 2s timeout (rust/op-reth/crates/txpool/src/interop_filter/client.rs:44 DEFAULT_REQUEST_TIMEOUT, no retry). A blocked verdict times out, counts as a non-response, and with endpoints=1 / min_responses=1 (the devstack/in-process setup) quorum can't be reached, so the pool fails closed and rejects all interop txs until the filter answers again.

Observed

CI job 5353712 (memory-all-opn-op-reth, develop, 2026-07-21): during a ~9.6s machine-wide scheduler stall, the single in-process filter endpoint gave no answer for 2s+, both chains' op-reth logged inbox entry validation timed out, timeout: 2 secs, and chain 902's pool logged interop failing closed: too few endpoints returned a definitive verdict to reach quorum (received=0 required=1 endpoints=1). Full narrative in #21940.

The design intent is that ingestion lag degrades to a soft "data not yet available" verdict (which op-reth treats as a non-response without failing closed on its own) — not to silence.

Proposed fix

Don't answer verdicts under the ingestion write lock:

  • narrow processBlockLogs's critical section (e.g. stage logs outside the lock, hold it only for the actual DB mutation), and/or
  • serve the read path from a snapshot / atomically-published latest-seal, so a slow ingest yields a fast ErrFuture-style soft response instead of a blocked RPC.

Cross-ref: #21940 (flake where this surfaced).

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: bugs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions