feat(state): stitch tree_aux root serving across the vct upgrade height

[p0mvn]

Motivation

ReadRequest::BlockRoots served per-block commitment roots with an either-or: if the commitment_roots_by_height index had anything at start_height it returned only the index's contiguous prefix, otherwise it derived from the per-height trees. It never stitched the two sources across a boundary.

Because the index is written on both commit paths, the only boundary that can exist is the upgrade height U — the first height the new binary commits: below U a DB written by the old binary has per-height trees but no index; at/above U the index is present. A peer request straddling U (e.g. [x, x+500] with U = x+100, default vct mode above so trees stop at U) returned only the ~100-root prefix, which the fetch client rejects under its minimum-progress threshold (¼ of count) — a permanent stall (state.vct.root.stalled.height). This addresses the TODO previously sitting in the BlockRoots handler.

Solution

• New one-time vct_upgrade_height marker U in a vct_upgrade_metadata column family — the lowest height this binary commits (and the lowest height in the serving index). Written once on the first committed block (both commit paths) and never moved.
• Serving stitch (serve_block_roots): per-height trees below U + serving index at/above U, concatenated into one gap-free run. A pre-index archive node (no U) still derives the whole range from the trees.
• Band-aware tree availability: trees are absent only within [U, H) (H = checkpoint handoff), via a new vct_tree_absent helper. Trees are present below U (pre-upgrade) and at/above H (semantic sync). The z_gettreestate gate and the sapling/orchard_tree_by_height guards now use the band. For a genesis fast-sync (U = 0) this reduces exactly to the prior height < H behaviour.
  • When the upgrade is at/after the handoff (U >= H) the band is empty, so every height is servable — matching the design (semantic sync keeps writing trees above the checkpoint).

Tests

• Unit tests for vct_tree_absent (the [U, H) band, plus the empty-band U >= H case) and an index-side serve_block_roots test.
• End-to-end stitch assertion added to the DB-produced round-trip proptest: simulates a mid-chain upgrade by dropping the serving index below U, and asserts serve_block_roots still returns the full range gap-free, matching the all-trees reference.
• U = 0 assertion in the genesis fast-sync proptest.
• Green: new tests, zebra-state vct/commitment/tree suite (except one pre-existing failure, see below), zebra-network -- tree_aux, zebra-rpc -- treestate. cargo fmt/clippy clean for the changed code.

Specifications & References

verified-commitment-trees design (§4 serving index, §9 tree_aux serving).

Follow-up Work

• Pre-existing (not from this PR): vct_frozen_frontier_survives_reopen fails on the base branch independently of these changes — the test's custom network has no embedded frontier, so reopening a frozen DB panics at FinalizedState::new. Confirmed it fails identically with this PR's marker write disabled, and for its genesis fast-sync (U = 0) vct_tree_absent is byte-identical to the prior guard.
• Conservative over-rejection is intentional: toggling checkpoint_verify=false mid-run backfills trees only above the current tip, so [U, H) keeps reporting unavailable; full recovery is a fresh reindex from genesis.

AI Disclosure

• AI tools were used: Claude Code for the analysis, implementation, and tests in this PR (reviewed by the author).

PR Checklist

• The PR title follows conventional commits format.
• The PR follows the contribution guidelines.
• This change was discussed in an issue or with the team beforehand.
• The solution is tested.
• The documentation and changelogs are up to date.

Consensus-sensitive: touches the finalized commit and serve paths. Opened as a draft for human review.

[v12-auditor]

Warning

Insufficient credits for auto-review. Keep at least $5.00 of available balance to start a run. Please add credits to continue.

[p0mvn]

Note: this was the main bug I suspect