Reorg-canceled batch leaves a permanent id gap that panics on restart

[biryukovmaxim]

Summary

A reorg that cancels an in-flight tip batch permanently bricks node restart.

ScheduledBatch::commit (scheduling/scheduler/src/scheduled_batch.rs:386-387) is gated on if !self.canceled(), so a canceled batch never writes its StateSpace::BatchMetadata key, while CanonicalChainManager::rollback (storage/canonical-chain/src/manager.rs:63-66) only rolls the canonical-bit oracle back and never pops entries. The canceled id stays allocated, the next block takes a higher one, and an interior hole is left in the BatchMetadata column family.

Nothing backfills it: Rollback::execute only repoints latest pointers, and scheduling/scheduler/src/pruning_worker.rs:231 only deletes below the pruning bound.

Impact

On every subsequent start, Store::canonical_chain_manager (storage/types/src/store.rs:33-43) replays the non-contiguous (id, metadata) pairs, push (manager.rs:97-102) re-densifies them onto base + entries.len(), and the tip's metadata() lookup returns None, panicking at .expect("walked id is live") (manager.rs:110). This happens in release too.

No adversary is required and no trigger is exotic: a reorg landing while the tip batch is in flight is routine. Recovery requires manual DB surgery.

Reproduction

Repro branch:

https://github.com/biryukovmaxim/vprogs/tree/repro/g5-reorg-canceled-batch-id-gap

Focused command (must run with debug assertions off):

cargo test --release -p vprogs-scheduling-scheduler --test reorg_id_gap -- --nocapture

Observed result on current master, 20/20 runs:

thread 'canceled_tip_batch_strands_an_id_and_bricks_restart' panicked at storage/canonical-chain/src/manager.rs:110:44:
walked id is live

The test persists ids [1, 3], strands id 2 interior, avoids a same-hash flip-reorg, and asserts the node fails to restart.

Why debug assertions must be off

The debug_assert_eq!(assigned, id, "restore must be contiguous") at manager.rs:36 catches the non-contiguous replay and aborts before the restart path runs, demonstrating a different failure than the one shipped to users:

panicked at storage/canonical-chain/src/manager.rs:36:13:
assertion `left == right` failed: restore must be contiguous
  left: 2 / right: 3

The root Cargo.toml sets no [profile.release] debug-assertions override, so production builds hit the ancestry-walk panic instead. The repro asserts the profile up front and fails fast rather than silently demonstrating the wrong bug.

Scope

• Proves the defect at the scheduler/storage layer, via Scheduler::rollback_to on a real RocksDB store. The brick is permanent and deterministic once the gap exists.
• Does not prove that a real L1 reorg drives this path in production: rollback_to is driven directly, not via a live reorg.
• The test gates the cancellation to make it deterministic. The natural timing window is real (9/10 ungated) but was not measured under production load.

Suggested Fix

Either pop the canceled id in rollback, or make restore tolerate an interior gap. The repro passing is the acceptance criterion.