Evan/perf plus download fixes#261
Closed
p0mvn wants to merge 36 commits into
Closed
Conversation
This reverts commit 3314496.
…ot rejection The reactor already rejects header-sync responses that carry tree-aux roots on a non-finalized range (`UnrequestedTreeAuxRoots` at decode, `MalformedMessage` at the reactor), but the tests still sent roots on those ranges. Switch non-finalized test messages to roots-free builders, add the `finalized_*` opt-in builders for finalized ranges, and add guard tests: - `decode_rejects_tree_aux_roots_when_not_requested` - `non_finalized_response_carrying_tree_aux_roots_is_malformed` Also wait on the backfilled headers landing (not the pre-set finalized height) before asserting the backward checkpoint-range commit trace.
…hits
Add observability for the header-carried tree-aux-roots feature:
- header-sync `headers_served`/`headers_received` rows now carry
`want_tree_aux_roots` and `tree_aux_roots_len`, and the header-sync
driver's commit-state rows carry `tree_aux_roots_len`
- new `state.vct.fast_path.{hit,miss}` counters record whether a finalized
commit consumed peer-supplied roots to skip the note-commitment rebuild
- `insert_bool` trace helper and a small `root_bytes` if/else cleanup
…ock frontier reads A successful checkpoint commit no longer issues a per-block read-state frontier query; it returns `local_frontier: None` and the durable frontier is published by a coalesced periodic `CheckpointFrontierRefresh` instead. - short-circuit the post-commit frontier read on successful checkpoint applies (driver returns `local_frontier: None`) - rework `CheckpointFrontierRefresh` from `highest_sent` to a `baseline_verified_tip` baseline, adding `observe_verified_tip` so resets keep the baseline current; the driver's `query_checkpoint_refresh_frontiers` reads the finalized + non-finalized tips with split read-timing traces - attribute commit throughput from the apply RESULT (carrying body `bytes` on `SubmitItem`/`SubmittedBlockApply`) so a commit whose `applying` entry is reaped by a coalesced refresh before its completion drains is still counted exactly once - add commit-progress observability: `block_commit_progress` rollups, per-block `block_apply_finished` latency rows, and a `sync.block.submit.throttled` counter for apply-pipeline-limited submits
- drop an unused block_hash binding in a block-sync driver test - wrap a fully-const static-limits assertion in a const block - allow too_many_arguments on drive_zakura_header_sync_actions
…aders tree_aux_roots_for_served_header_range returned an empty vec whenever the available roots did not cover every requested header height. Because served headers normally run ahead of committed/provisional roots, that empty-on-gap behavior meant no roots were ever served over the header-sync path, silently disabling header-carried tree-aux roots. Stop at the first missing or misaligned height and return the aligned prefix collected so far.
…w block-sync
A standalone xtask-driven benchmark for the block-commit stack *below* Zakura
block-sync: the real checkpoint verifier + real finalized state committing real
mainnet blocks at production apply concurrency (401), isolated from networking.
- `fetch`: download blocks (and optional `z_gettreestate` roots) from a node's
JSON-RPC into an on-disk cache (resumable, validated).
- `run`: replay cached blocks through `CheckpointVerifier` + `zebra_state`,
reporting blk/s, MiB/s, and commit-latency percentiles; emits
`block_sync.jsonl` rollups. `--frontier-read {per-block|coalesced|none}` A/Bs
the coalesce change on the real stack; `--with-roots` feeds tree-aux roots into
the VCT fast path and verifies via `state.vct.fast_path.hit/miss`;
`--state-dir` hydrates from a state snapshot to reach a post-NU5 range.
- `snapshot`: download/verify/extract a Zebra state snapshot into
`~/.zakura/snapshots`.
- Profiling reuses the existing `profiling` cargo profile (samply) and
`jemalloc-profiling`.
Includes an offline smoke test that commits bundled mainnet blocks 0..=10
through the real verifier+state. `cargo xtask zakura-commit-bench` forwards args
to the crate.
* fix(network): bound Zakura apply-window drain Keep block download progress close to the checkpoint commit frontier so a full apply pipeline does not consume the shared byte budget with far-future bodies. * test(network): use limited Zakura drain helpers directly Avoid test-only unbounded wrapper methods by passing explicit unlimited limits at the call sites that need legacy behavior.
Header-carried tree-aux roots (the header-sync Headers message plus CommitHeaderRange persistence to zakura_header_commitment_roots_by_height) fully replace the old separate tree_aux request/response stream, so remove it. - zebra-network: delete the zakura/tree_aux client/server module, its stream kind, capability registration, and the tree_aux_port plumbing through init_with_zakura_header_sync / spawn_zakura_endpoint / service_registry. - zebrad: delete tree_aux_driver (StateTreeAuxPort serving + run_tree_aux_driver fetch loop) and its start.rs wiring; drop the now-dead tree_aux_roots_writer argument from drive_zakura_header_sync_actions. - zebra-state: remove TreeAuxRootsWriter, PeerSourceWriter, PeerSourceHandle, and the peer-root refetch signal; drop the writer from zebra_state::init's return. PeerSource stays as the DB-backed reader the committer uses. A missing or rejected root now waits for header sync to deliver a replacement via the in-place commit retry, or (on an archive node) recomputes from the per-height trees; it no longer refetches over a separate stream. zakura-commit-bench --with-roots is disabled pending a re-port onto the CommitHeaderRange path.
Record a one-time `vct_upgrade_height` marker `U` (the lowest height this binary commits, and the lowest height in the `commitment_roots_by_height` serving index) in a new `vct_upgrade_metadata` column family. Written once on the first committed block and never moved. Root serving (`ReadRequest::BlockRoots`) now stitches the per-height trees below `U` with the serving index at and above `U`, so a node that upgraded mid-chain serves a range crossing `U` as one gap-free batch instead of the short index-only prefix that stalled the fetch client's minimum-progress check. A pre-index archive node (no `U`) still derives the whole range from the trees. Historical note-commitment tree availability is now the band `[U, H)` (H = checkpoint handoff) via a new `vct_tree_absent` helper: trees are present below `U` (pre-upgrade) and at/above `H` (semantic sync), absent only in between. For a genesis fast-sync (`U = 0`) this reduces exactly to the prior `height < H` behaviour.
…rd for submitting non-contiguous blocks
|
Warning Insufficient credits for auto-review. Keep at least $5.00 of available balance to start a run. Please add credits to continue. |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…to evan/perf-plus-download-fixes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Made agaisnt #189