feat(docker): deterministic full-source-bootstrapped build via StageX - #13
Open
y4ssi wants to merge 23 commits into
Open
feat(docker): deterministic full-source-bootstrapped build via StageX#13y4ssi wants to merge 23 commits into
y4ssi wants to merge 23 commits into
Conversation
…empool_mainnet pass
…edBlock, SemanticallyVerifiedBlock, and CheckpointVerifiedBlock; calculate it on demand (#10392) * remove deferred_pool_balance_change from ContextuallyVerifiedBlock and SemanticallyVerifiedBlock * calculate deferred_pool_balance_change when finalizing a block (on demand) * update tests * fix: linting errors * fix: take deferred_pool_balance_change from caller in with_block_and_spent_utxos * test: add regression tests for deferred_pool_balance_change calculation * fix: update call to chain_value_pool_change in chain_value_pool_change_propagates_transaction_value_balance_errors * test: add regression tests for deferred_pool_balance_change calculation (and fix merge conflicts)
* fix(network): stop poisoning inventory registry on transient errors Only mark inventory as missing for explicit notfound responses. Transport failures (timeouts, drops) no longer poison the routing registry. Combined with #131 FindBlocks registration + re-request. Co-Authored-By: Gustavo Valverde <g.valverde02@gmail.com> * feat(sync): source-aware block routing with contiguous-prefix guard Track which peer announced each block hash from FindBlocks/ExtendTips responses. Route block requests to the announcing peer via BlocksByHashFrom, falling back to inventory routing if the source is not ready. Force the first CONTIGUOUS_PREFIX_P2C hashes to reliable P2C routing so the checkpoint verifier always gets its next-needed block. Based on v5.0.0 + no-poison fix. No #131 (re-request is superseded by proper routing). Co-Authored-By: Gustavo Valverde <g.valverde02@gmail.com> * feat(sync): add block re-request for failed downloads (#5709) Re-queue blocks whose download failed with NotFound instead of silently dropping them. Bounded by MAX_BLOCK_REOBTAIN_RETRIES. Combined with source-aware routing, this handles the case where the source peer is busy and the fallback misroutes — the block gets another chance via normal routing. Co-Authored-By: Zaki Manian <zaki@manian.org> * feat(sync): match #129 behavior — prefix=2000, batch by source peer Three fixes to match #129's proven 0-error genesis-to-tip result: 1. CONTIGUOUS_PREFIX_P2C: 32 → 2000 (most blocks use reliable P2C) 2. Batch blocks by source peer (SOURCE_BLOCK_BATCH_LIMIT=8) 3. Unsourced blocks dispatched individually via P2C Co-Authored-By: Gustavo Valverde <g.valverde02@gmail.com> * chore: remove unused download_and_verify_from and dead variable * fix(ci): allow large_enum_variant on pre-existing Setup enum The Setup enum in inbound.rs has a 224-byte Initialized variant vs 16-byte Pending variant. This is pre-existing and unrelated to the sync fix, but clippy with --all-features -D warnings now flags it. * docs: clarify re-request intent and batch dispatch semantics Address review feedback: - Clarify that re-requests go through P2C to avoid re-asking the NotFound peer (Zaki point C) - Fix misleading doc comment on download_and_verify_batch — each block gets its own readiness check, not a single shared one (Copilot #1) * refactor: remove dead source-aware routing code, keep behavioral fix Remove BlocksByHashFrom, source_by_hash tracking, route_source(), CONTIGUOUS_PREFIX_P2C, SOURCE_BLOCK_BATCH_LIMIT, and download_and_verify_batch. These were unreachable under any default config (prefix 2000 > max batch size in all modes). Keep the three behavioral changes that produce the 0-error result: - No-poison: stop registry poisoning on transient errors (client.rs) - Re-request: re-queue NotFound blocks instead of dropping (sync.rs) - Duplicate-tolerant dispatch: catch DuplicateBlockQueuedForDownload and continue instead of dropping the batch (sync.rs) 3 files, +128/-31 vs main. --------- Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com> Co-authored-by: Zaki Manian <zaki@manian.org>
fix(state): fix dequeue_children by_height removal logic
* feat(rpc): Add subver, version, services, lastrecv, banscore, and connection_state to getpeerinfo Extend the getpeerinfo RPC response with fields that help identify peer types and diagnose connectivity issues: - subver: peer's user agent string (e.g. "/Zebra:4.3.0/", "/MagicBean:6.11.0/") - version: negotiated protocol version - services: offered services as zero-padded hex string - lastrecv: timestamp of last message received - banscore: misbehavior score - connection_state: current connection state To plumb user_agent and negotiated_version from the handshake into the address book, MetaAddr and MetaAddrChange had Copy removed (user_agent is a String). All downstream code updated to use Clone instead. * docs: Add changelog entries for getpeerinfo extensions Document the breaking changes and new public API additions in zebra-network (MetaAddr lost Copy, new_connected signature changed, new accessors and Version type) and zebra-rpc (PeerInfo::new signature changed, new field accessors). * fix(rpc): use stable Display strings for connection_state in getpeerinfo * test(rpc): update getpeerinfo snapshots for NU6.2 protocol version --------- Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
Three independent failures are currently red on main: - CHANGELOG.md had a duplicated "Zebra 4.3.0" heading, tripping markdownlint MD022 (blanks-around-headings). Removed the duplicate. - A zebra-rpc test imported the ParameterDifficulty trait as `_` even though it is already in scope via `use super::super::*`. Newer nightly flags the redundant import, failing cargo-udeps. Removed it. - proc-macro-error2 is unmaintained (RUSTSEC-2026-0173). Added it to the deny.toml advisories ignore list next to its predecessor.
…692) * test(network): add regression test for INITIAL_MIN_NETWORK_PROTOCOL_VERSION (#10682) Assert that the startup-time minimum peer protocol version floor matches the latest settled network upgrade. This test currently fails because the floor is pinned to Nu6 (170_120) while the latest settled upgrade is Nu6_2 (170_150). * fix(network): bump INITIAL_MIN_NETWORK_PROTOCOL_VERSION to Nu6_2 (#10682) The startup-time minimum peer protocol version floor was pinned to Nu6 (170_120 on Mainnet) but the latest settled upgrade is NU6.2 (170_150). This caused the node to accept peers with stale protocol versions during the brief pre-tip window at cold start. Closes #10682 * fix(network): remove flawed min-protocol-version test, add release checklist item
…t (#10702) fix(test): retry block proposal on tip race in getblocktemplate test
* refactor(ci): centralize path filters and align check naming Replaces the per-workflow detect-changes composite with dorny/paths-filter v4.0.1 reading a shared path-filters.yml. The previous regex literals were duplicated across seven workflows and prone to drift. Each aggregator job is renamed to match its workflow basename, so the job ID, the workflow file, and the required-check context all share one name (lint, unit-tests, test-crates, test-docker, book, coverage, docs-check). A grep on the failing context name lands on the producing workflow in one hop. merge_group is added to every required-check workflow so the queue migration can land without re-touching CI config. book.yml and coverage.yml stay push-only because they deploy rather than validate. The deleted composite also under-resolved the diff base: it used github.event.pull_request.base.sha instead of the merge-base, which over-triggered when main advanced under a long-lived PR. BREAKING CHANGE: the required-check contexts are renamed from 'lint success', 'test success', 'test crate build success' to 'lint', 'unit-tests', 'test-crates'. Update the GitHub ruleset in the same window this PR merges.
…failure (#10701)
queue_conditions previously only checked '#approved-reviews-by >= 1'
and relied on Mergify auto-mirroring branch protection. That mirror
works for classic branch protection but not GitHub Rulesets, so
failing required checks and changes-requested reviews were not
respected by the queue.
Add two explicit gates:
- '#changes-requested-reviews-by = 0'
- check-success against the three aggregator check runs ('lint', 'unit-tests', 'test-crates')
* fix(zebrad): reduce end of support for NU7 network upgrade Reduce EOS_PANIC_AFTER from 105 days to 44 days and update ESTIMATED_RELEASE_HEIGHT so nodes stop running before the NU7 network upgrade expected at the end of July 2026. - Warning starts: ~July 11 (30 days after release) - Panic (stops running): ~July 25 (44 days after release) Closes #10709 * chore(supply-chain): trust oxarbitrage as publisher for zebra crates Add cargo-vet trust entries for all 9 zebra crates published by oxarbitrage, replacing version-pinned exemptions. This fixes the cargo-vet check failure after the v5.1.0 crate publish and prevents it from recurring on future releases. * chore(supply-chain): trust all team members as zebra crate publishers Add cargo-vet trust entries for gustavovalverde, conradoplg, upbqdn (Marek), arya2, and natalieesk so any team member can publish crate releases without cargo-vet failures. Pili not included (no crates.io account found).
GitHub Actions does not pass repository secrets to workflows triggered by pull_request events from forks. As a result, Z3_APP_ID and Z3_APP_PRIVATE_KEY resolve to empty strings on fork PRs, and actions/create-github-app-token rejects the empty app-id, failing the job on every external contribution. Gate the trigger-integration job on the same fork-skip pattern already used by the zfnd-* workflows (e.g. zfnd-deploy-nodes-gcp.yml). The workflow still runs on same-repo PRs and main pushes; only fork PRs are skipped, removing the perma-failing required check from external contributors.
Syncs the fork up to upstream main (990dcb6), bringing the fork's 4 local commits (docs repoint to zodl-inc, .cargo/config.toml cstdint fix) together with 21 new upstream commits. Sync for Linear COR-1293 (ZODL fork of Zebra + release/deploy infra). # Conflicts: # README.md # book/src/dev/continuous-integration.md
Port the StageX deterministic build for zebrad from upstream PR ZcashFoundation/zebra#10491 (authored by Anton Livaja / @antonleviathan, Distrust) onto the zodl-inc fork, plus the hash-bump + preflight helpers from the earlier PR #10068. Files: - docker/Dockerfile.deterministic: 4-stage (deps/tests/release/runtime) build on SHA-pinned StageX bases. Static musl (x86_64-unknown-linux-musl, +crt-static), codegen-units=1, SOURCE_DATE_EPOCH=1, --build-id=none, clang+lld linker, libstdc++->libc++ linker-script shim for librocksdb-sys and libzcash-script. Runtime image is FROM scratch + busybox. - docker/build.sh: OCI + binary export driver with a compat.sh preflight. - docker/compat.sh: checks Docker 26+, buildx 0.13+, containerd image store. - docker/update-stagex.sh: bumps the pinned StageX digests in place. - docker/README.md: rationale, requirements, provenance, verify steps. - docker/stagex-reference/: vendored pallet-rust + core-rust definitions and a digest table so reproduction inputs are legible without cloning StageX. - .gitignore: ignore /build/ (deterministic build output). Adaptation notes vs PR #10491 (written against an older main): - Workspace crate list (12 members) verified unchanged against the synced tree's Cargo.toml, so the release/tests COPY lists are correct as-is. - StageX pallet-rust is pinned to 1.94.0 and is the toolchain we actually build with. The fork's standard Dockerfile uses RUST_VERSION=1.91.0 and the workspace MSRV is 1.85.1; 1.94.0 satisfies both. ARG RUST_VERSION is left documentary (real version comes from the pinned image) and aligned to 1.91.0. The StageX bases are pinned verbatim from the upstream PR as the reproducibility source of truth. Linear COR-1293.
|
PR titles must follow Conventional Commits format. Your title needs a small adjustment. See the contribution guide for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a reproducible (StageX full-source-bootstrapped) Docker build for zebrad. Ported from upstream PR ZcashFoundation/zebra#10491 (Anton Livaja / Distrust) + the compat/update helpers from #10068, adapted to this fork.
Verified reproducible
Built three times on a clean host (incl. a full recompile after
buildx prune -af) — all three OCI tarballs are bit-identical:The determinism knobs (
SOURCE_DATE_EPOCH=1,codegen-units=1, static musl,--build-id=none,rewrite-timestamp=true) all hold across a from-scratch recompile.What's included
docker/Dockerfile.deterministic— 4-stage StageX build (SHA-pinned bases: pallet-rust 1.94.0, pallet-clang, core-busybox, user-protobuf/abseil, core-gmp/mpfr/mpc/isl), libstdc++->libc++ linker shim for RocksDB/zcash-script.docker/build.sh— OCI output driver with reproducible flags.docker/compat.sh— Docker 26+/buildx/containerd preflight.docker/update-stagex.sh— StageX base digest bumper.docker/stagex-reference/— vendored pallet-rust + core-rust Containerfiles for legibility..gitignore—/build/.Notes
Depends on: the
sync/upstream-main-2026-06-15PR landing first.