Enable state pruning for archival nodes

[imabdulbasit]

Currently the archival node store all the derived and underived data such as merklized state. A node only needs to keep the most recent merklized state data to serve catchup requests, and this data can be derived by replaying the leaves and applying the state transition function.

This PR adds a state garbage collector for the archival nodes that deletes state more than ESPRESSO_NODE_ARCHIVE_STATE_RETENTION heights behind the latest state height. Set ESPRESSO_NODE_ARCHIVE_FULL_STATE to retain everything as before. Nodes running the pruner are unaffected, since it already prunes state alongside the other data, and archive startup no longer clears the state pruned height.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[sveitser]

@imabdulbasit what's the point of making it configurable then? If it's useless we should just always delete it.

[imabdulbasit]

I would like to keep it optional at first, so if anything goes wrong then we can just disable it via an env variable. We hav had issues before with pruning so I would prefer to keep it optional

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[claude]

Claude finished @imabdulbasit's task in 9m 7s —— View job

---

Code review

• Gather context (diff, existing comments, surrounding pruner/backfill code)
• Review ArchiveStateGc logic
• Review config/options changes
• Review hotshot-query-service changes
• Review tests & docs
• Post review (6 inline comments)

---

The design is sound and the reasoning in the doc comments is genuinely good — especially the delete + save_state_pruned_height in a single prune transaction, which is stronger than the pruner's two-step (prune_batch commits the height first, then deletes, leaving a window where the height claims data is gone before it is). Keeping the state pruned height on archive startup is also a real fix, not just a convenience: state is never re-fetched from peers, so the old DELETE FROM pruned_height left archive nodes serving commitment-mismatch errors for heights whose rows a previous pruning run had already collapsed.

I verified the underlying deletion invariant holds: delete_state_batch keeps the newest version at or below to for every path, and everything above to is untouched, so any snapshot above the pruned height remains fully reconstructible. The batching, the resume-from-pruned-height behaviour, and the from >= target early return are all correct.

Six inline comments. The ones I'd act on before merge:

1. batch_size == 0 underflows (api/sql.rs:176) — min(from + 0, target) - 1 wraps to u64::MAX in release, which deletes all non-newest versions including inside the retention window, records pruned_height = u64::MAX, and then loops forever. Reachable via ESPRESSO_NODE_PRUNER_BATCH_SIZE=0, because ArchiveStateGc::new builds PrunerCfg::from(opt.pruning) without ever calling validate() (and validate() doesn't check batch_size anyway — archive mode skips Config::pruner_cfg() entirely since --prune conflicts with --archive). One-line clamp suggested inline. Fix this →

2. Does this actually reclaim the disk it claims? (api/sql.rs:197) — hash_bigint is append-only and nothing ever deletes from it, so every deleted merkle node leaves its hash row (plus the UNIQUE index on value) behind. Roughly one orphan per deleted node version. Would be good to state whether the measured savings account for this, even if the orphan sweep is follow-up work.

3. Fights the *_bigint backfill on Postgres (api/sql.rs:184) — state_tables() only lists the *_bigint tables, and BackfillFeeMerkleTree/BackfillBlockMerkleTree move legacy rows up by ascending created, so on a node mid-backfill the GC deletes rows the backfill then re-inserts below the pruned height, where nothing can read them.

4. User-facing behaviour change worth calling out explicitly. After collection, get_path returns NotFound for any snapshot <= state_pruned_height. That takes out more than catchup: the header-proof endpoint (api/light_client.rs:360, Snapshot::Index(root) with a caller-chosen root) and the fee/reward account-proof endpoints stop serving roots older than the window — ~16 days at the 700k default. Archive nodes are exactly what integrators point at for historical proofs, so this deserves a release note, and it's the substantive counterpoint to @sveitser's thread: the data isn't useless, it's just expensive, and this PR doesn't ship a way to re-derive a specific old snapshot on demand.

Testing gaps (existing tests are well constructed — the "newest version at the pruned height survives" assertion is exactly the right thing to pin):

• Nothing covers the gating in api/options.rs:583 — that the task is spawned only when archive && !archive_full_state.
• Nothing pins DELETE FROM pruned_height WHERE id = $1 in hotshot-query-service (suggested inline).
• Only the fee tree is exercised; block_merkle_tree_bigint is the other configured state table and is untested here.
• Edge cases not covered: head < retention (no-op via target == 0) and retention == 0.

CI: the two failures reported earlier (test_archive_state_gc_batches, config_node_response_snapshot) look addressed by d67a430. Test, Build, Slow Test, HotShot integration tests and Cargo features are still running on the head commit — worth confirming green, particularly test_archive_state_gc_batches, which failed twice within a single run.

Nits: stray blank line between the doc comment and #[clap(...)] (persistence/sql.rs:257); the GC's pacing silently comes from ESPRESSO_NODE_PRUNER_* options that --archive otherwise disables, which should be documented; ArchiveStateGc::vacuum duplicates SqlStorage::vacuum, and save_state_pruned_height now has two implementations (Transaction<Write> via upsert, Transaction<Prune> via raw SQL); --archive-full-state is silently ignored without --archive and could carry requires = "archive".

No security concerns — no new external input reaches these queries, table names come from compile-time state_type() constants, and the widened visibility (prune_write, delete_state_batch) stays inside the workspace.
· branch abdul/archive-state-pruning

[claude]

Underflow / infinite loop / over-deletion when batch_size == 0.

ESPRESSO_NODE_PRUNER_BATCH_SIZE=0 is accepted here: PrunerCfg::validate() doesn't check batch_size, and ArchiveStateGc::new builds the cfg with PrunerCfg::from(opt.pruning) (line 141) without ever calling validate() — archive mode never goes through Config::pruner_cfg(), since --prune conflicts with --archive.

With batch_size == 0 and pruned == None (from == 0), min(0 + 0, target) - 1 underflows. In release builds that wraps to u64::MAX, and then:

• delete_state_batch(tables, u64::MAX) deletes every row that isn't the newest at its path — collapsing all history including inside the retention window, which breaks catchup for any height below head;
• save_state_pruned_height(u64::MAX) records that;
• from = to + 1 wraps back to 0, so the while from < target loop never terminates.

Cheap fix — clamp the batch and avoid the bare subtraction:

Suggested change

	let to = min(from + self.cfg.batch_size(), target) - 1;
	let batch_size = self.cfg.batch_size().max(1);
	let to = min(from + batch_size, target) - 1;

[claude]

Interaction with the in-flight *_bigint backfill.

cfg.state_tables() is ["block_merkle_tree_bigint", "fee_merkle_tree_bigint"] (persistence/sql.rs:664). The legacy fee_merkle_tree / block_merkle_tree tables are still populated on Postgres nodes that haven't finished BackfillFeeMerkleTree / BackfillBlockMerkleTree, and those backfills move legacy rows into *_bigint in ascending created (persistence/migrations.rs:104-155).

So on a node mid-backfill:

• the GC deletes old *_bigint rows and advances the state pruned height;
• the backfill then re-inserts rows for those same old created values (ON CONFLICT (path, created) DO NOTHING no longer conflicts, since the row is gone);
• those resurrected rows are below the pruned height, so no read will ever return them — they're invisible dead weight until the next GC interval deletes them again.

Not a correctness bug (reads at created <= pruned_height return NotFound), but it's wasted I/O and bloat, and no disk is actually reclaimed until the backfill passes the retention watermark. Worth either gating the GC on those two backfills being complete, or including the legacy tables in the delete list while they exist.

[claude]

Orphaned hash_bigint rows cap the space actually reclaimed.

Merkle node rows reference hash_bigint(id) (FK ... REFERENCES hash_bigint(id), V1501__hash_bigint_expand.sql), and nothing in the codebase ever deletes from hash_bigint — it's append-only, deduplicated by value with a UNIQUE index. Since almost every node version has a distinct hash, hash_bigint holds roughly one row per deleted node version, so its row count and bytes (8-byte id + 32-byte value + PK + unique index on value) are in the same ballpark as the merkle rows this GC removes.

Net effect: the deletion is real but the disk win is likely a fraction of what the PR description implies. A batched sweep of orphans, e.g.

DELETE FROM hash_bigint AS h
WHERE h.id IN (SELECT id FROM hash_bigint WHERE id < $watermark LIMIT $batch)
  AND NOT EXISTS (SELECT 1 FROM fee_merkle_tree_bigint   WHERE hash_id = h.id)
  AND NOT EXISTS (SELECT 1 FROM block_merkle_tree_bigint WHERE hash_id = h.id)

would close the gap. Same gap exists for the time-based pruner, so it's arguably follow-up work — but it's worth stating explicitly in the PR whether the measured savings account for it.

Minor, in the same function: this duplicates SqlStorage::vacuum (hotshot-query-service/src/data_source/storage/sql.rs:1319); since prune_write was just made pub, making vacuum reusable (taking pages as an argument) would avoid the second copy.

[claude]

On an existing archive node the first collect() call walks the entire history in one pass — with head in the millions and the default batch_size of 1000 that's thousands of delete transactions back to back, and on SQLite vacuum() only runs after the whole backlog finishes, so no space is reclaimed until then (and the freelist grows the whole time). The created index keeps each individual batch cheap, so this is a pacing/observability point rather than a query-plan one: consider vacuuming every N batches and logging progress (rows deleted / current from) so operators can see where a multi-hour first pass is.

Also, unlike the pruner (fetching.rs:443-470) there's no per-batch error backoff. Not a bug — a failed run resumes from the recorded pruned height on the next interval — but a transient error costs a full interval() (1.5h by default) of progress.

[claude]

Stray blank line between the doc comment and the attribute (it compiles, but it's inconsistent with every other option here). Also worth documenting that the GC's pacing comes from the pruner options — ESPRESSO_NODE_PRUNER_INTERVAL (default 1.5h) and ESPRESSO_NODE_PRUNER_BATCH_SIZE — which is surprising given --archive conflicts with --prune:

Suggested change

	/// Number of recent block heights of merklized state an archive node retains.
	///
	/// Collection runs on the pruner interval (PRUNER_INTERVAL) in batches of PRUNER_BATCH_SIZE
	/// heights, even though the pruner itself is disabled in archive mode.
	#[clap(

[claude]

Good change, and it also fixes a latent bug: previously an archive restart cleared the state pruned height too, but merklized state is never re-fetched from peers (only consensus data is), so get_path at those heights would stop returning NotFound and instead reconstruct from surviving nodes — yielding a commitment mismatch error, or wrong data if the commitment check were absent.

Worth a regression test alongside the existing pruning tests in this file: connect with archive, having seeded both pruned_height rows, and assert load_pruned_height() == None while load_state_pruned_height() == Some(h). Nothing currently pins the WHERE id = $1, so a future refactor could silently restore the old behaviour.