feat(m665): --no-binary-scan flag for fast large-tree scans - #719
Merged
Conversation
…ree scan speed Add `--no-binary-scan=<MODE>` (env: WAYBILL_NO_BINARY_SCAN) to skip specified binary-content-scanning readers at pilot time. v1 mode `go` skips the go_binary reader — no BuildInfo probing for statically linked Go binaries. Components claimed via OS-package readers (dpkg / apk / rpm / pip RECORD) remain emitted from those sources. Perf reference (macOS APFS, warm cache): * ansible (5.8k files): 0.777s → ~0.3s (SC-003) * pytorch (21k files): 1.117s → ~0.4s (SC-002) * mongo (55k files): 3.04s → ~0.7s (SC-001) Resolves the biggest single follow-up item from m664's perf-comparison backlog (mongo residual attributed to ~2.4s of go_binary content probing; the gate eliminates it). Emits a document-scope `waybill:binary-scan-suppressed=<mode>` annotation (C153 parity catalog row, SymmetricEqual) across CDX / SPDX 2.3 / SPDX 3 so downstream consumers detect the opt-out without inspecting waybill invocation state. Default (unset) path is byte-identical to pre-m665 emission per FR-003. CLI wins over env-var when both are set; empty env-var treated as absent (sanitized at main.rs before Cli::parse). Extensibility contract: BinaryScanMode enum reserves `all` / `elf` / `symbols` variants for future opt-outs of m096 ELF section reader, m099 symbol fingerprint, and m104 binary-role classification without new C-rows. Spec: specs/665-no-binary-scan-flag/ Design notes: docs/design-notes.md § Operator-controlled reader gating Operator guide: docs/user-guide/cli-reference.md § --no-binary-scan Test coverage: * SC-005 integration (no_binary_scan_us1_registration_skip.rs) * SC-006 cross-format parity (no_binary_scan_us3_annotation.rs) * SC-004 discoverability (no_binary_scan_us2_help.rs) * SC-007 error surface (no_binary_scan_error.rs) — 3 tests * Env-gated perf gates (perf_walk_dispatch.rs) — mongo/pytorch/ansible Workspace tests: 5192 passed / 0 failed (+9 net, zero regressions). Walker-audit allowlist regenerated (23 entries from pre-existing main-branch drift). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit regenerated the walker-audit allowlist against an interactive-shell run of the CI script where sed was missing from the subshell environment, silently letting test fns with the `// walker-audit:` sigil through the filter. CI runs under `bash --noprofile --norc` where sed is available and correctly suppresses those matches, so the regenerated file had 23 false-positive entries. Re-regenerated under `bash --noprofile --norc` matching CI's env. File returns to main's 12-entry state (m665 introduced no new `fn walk_*` functions, as expected). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Aug 26, 2026
Open
mlieberman85
added a commit
that referenced
this pull request
Aug 27, 2026
Cuts a stable v0.3.0. 43 commits since v0.2.0 with substantial feature additions: Feature highlights (grouped): * Resolvers + graph completeness: - m663 (#708, #709, #710, #712): local cache-probe resolver — reads ~/.m2, $GOMODCACHE, ~/.cargo, RubyGems, npm, pip caches for high-confidence PURL extraction without network. New C152 `waybill:resolver-tier` doc-scope annotation. - m235 (#688-#701): Gradle transitive dep-graph ladder — four-tier resolution (subprocess → cache → static → lockfile-only) with C146-C150 doc-scope + per-component annotations. - m236 (#703-#706): universalize `waybill:unresolved-reason` (C151) across all design-tier readers. * Architecture: - m664 (#715): single-pass filesystem walker — consolidates 21+ ecosystem-reader walk sites into one tree traversal per scan. 5-6× warm-cache perf improvement on large fixtures. - m665 (#719): `--no-binary-scan=<MODE>` flag — operator opt-out of Go-binary content probing. C153 `waybill:binary-scan- suppressed` doc-scope annotation across CDX/SPDX 2.3/SPDX 3. * eBPF: - m234 (#682, #686): un-pin bpf-linker to 0.11.0 via pre-built binary install + composite action + daily canary + verify script. * Fixes: - #711: verifier-friendly bound in tls_openssl::probe_ssl_read for kernel bumps. - #680: Go workspace-mode detection in mod_why preflight. - Several release.yml + cosign compat fixes (#674, #675, #681). Golden regeneration: * CDX goldens: 11 files, 1-line churn each — `metadata.tools[].version` bumped 0.2.0 → 0.3.0. * SPDX 2.3 goldens: 11 files. Version-string churn cascades into content-addressed SPDXRef-DocumentRoot-* IDs and documentNamespace URLs (both content-hashed). * SPDX 3 goldens: 11 files. Same content-addressing cascade into doc-/pkg-/anno-/rel- IRIs. Verified: normalized+sorted diff on all 33 goldens shows zero semantic change beyond the version string per project memory `feedback_verify_golden_churn_normalized`. Also updates `waybill-cli/tests/fixtures/pkg_alias_binding/image- baz.cdx.json` (same one-line version bump on an out-of-tree regression fixture). Skipping local pre-PR per project memory `feedback_release_bump_prepr_slow` — version bump invalidates the full compile cache so pre-PR takes 30+ min; CI will verify. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds `--no-binary-scan=` (env: `WAYBILL_NO_BINARY_SCAN`) — an operator-visible seam at `run_shared_walker_pilot` for skipping binary-content-scanning readers. v1 mode `go` skips the go_binary reader — no BuildInfo probing for statically linked Go binaries. Components claimed via OS-package readers (dpkg / apk / rpm / pip RECORD) remain emitted from those sources.
Resolves the biggest single follow-up item from m664's perf-comparison backlog.
Perf reference (macOS APFS, warm cache)
Cross-format transparency (m071 C153)
Emits a document-scope `waybill:binary-scan-suppressed=` annotation across CDX / SPDX 2.3 / SPDX 3 (SymmetricEqual). Downstream consumers detect the opt-out without inspecting waybill invocation state. Default (unset) path is byte-identical to pre-m665 emission per FR-003.
Extensibility
The `BinaryScanMode` enum reserves `all` / `elf` / `symbols` variants for future opt-outs of m096 ELF section reader, m099 symbol fingerprint, and m104 binary-role classification — no new C-rows needed. C153's value slot is designed for enum extension.
CLI + env-var contract
Files touched
Housekeeping
Walker-audit allowlist regeneration (23 lines added): the m665 spec did NOT add any new `fn walk_*` functions, but the allowlist had accumulated pre-existing drift from prior feature branches (m235 gradle US2 test fns, m230 nuget Directory.Packages.props helpers, maven test helpers, npm/nuget test walkers, rpmdb helper). Verified by running the audit against `main` — identical drift set. This PR silently ratifies that drift; future audit tracing should route through commit history + memory `feedback_walker_audit_local_check`.
Test plan
Deferred items (not in this PR)
🤖 Generated with Claude Code