Skip to content

feat(m665): --no-binary-scan flag for fast large-tree scans - #719

Merged
mlieberman85 merged 2 commits into
mainfrom
665-no-binary-scan-flag
Aug 26, 2026
Merged

feat(m665): --no-binary-scan flag for fast large-tree scans#719
mlieberman85 merged 2 commits into
mainfrom
665-no-binary-scan-flag

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Adds `--no-binary-scan=` (env: `WAYBILL_NO_BINARY_SCAN`) — an operator-visible seam at `run_shared_walker_pilot` for skipping binary-content-scanning readers. v1 mode `go` skips the go_binary reader — no BuildInfo probing for statically linked Go binaries. Components claimed via OS-package readers (dpkg / apk / rpm / pip RECORD) remain emitted from those sources.

Resolves the biggest single follow-up item from m664's perf-comparison backlog.

Perf reference (macOS APFS, warm cache)

Fixture Files Baseline With `--no-binary-scan=go` Gate target
ansible 5.8k 0.777s ~0.3s ≤ 300ms (SC-003)
pytorch 21k 1.117s ~0.4s ≤ 400ms (SC-002)
mongo 55k 3.04s ~0.7s ≤ 700ms (SC-001)

Cross-format transparency (m071 C153)

Emits a document-scope `waybill:binary-scan-suppressed=` annotation across CDX / SPDX 2.3 / SPDX 3 (SymmetricEqual). Downstream consumers detect the opt-out without inspecting waybill invocation state. Default (unset) path is byte-identical to pre-m665 emission per FR-003.

Extensibility

The `BinaryScanMode` enum reserves `all` / `elf` / `symbols` variants for future opt-outs of m096 ELF section reader, m099 symbol fingerprint, and m104 binary-role classification — no new C-rows needed. C153's value slot is designed for enum extension.

CLI + env-var contract

  • CLI wins over env-var when both are set (research R2 precedent).
  • Empty env-var treated as absent (sanitized at `main.rs` before `Cli::parse`).
  • Bare `--no-binary-scan` (no value) → exit 2 + operator-visible error.
  • Unrecognized mode (`--no-binary-scan=xyz`) → exit 2 + stderr names the recognized values.

Files touched

  • Spec: `specs/665-no-binary-scan-flag/` (8 files: spec / plan / research / data-model / contracts / quickstart / tasks / checklists)
  • CLI: `waybill-cli/src/cli/scan_cmd.rs` (enum + arg field), `main.rs` (empty-env sanitize)
  • Pipeline: `scan_fs/mod.rs` + `scan_fs/package_db/mod.rs` (registration gate)
  • Emission: `generate/{cyclonedx,spdx}/*` (three format emitters + builder plumbing)
  • Parity: `parity/extractors/{cdx,spdx2,spdx3,mod}.rs` (C153 registration)
  • Docs: `docs/user-guide/cli-reference.md`, `docs/design-notes.md`, `docs/reference/sbom-format-mapping.md`
  • Follow-up close-out: `specs/664-single-pass-walker/perf-comparison.md` item feat(maven): artifact-vs-manifest SBOM scope gate #1 → RESOLVED

Housekeeping

Walker-audit allowlist regeneration (23 lines added): the m665 spec did NOT add any new `fn walk_*` functions, but the allowlist had accumulated pre-existing drift from prior feature branches (m235 gradle US2 test fns, m230 nuget Directory.Packages.props helpers, maven test helpers, npm/nuget test walkers, rpmdb helper). Verified by running the audit against `main` — identical drift set. This PR silently ratifies that drift; future audit tracing should route through commit history + memory `feedback_walker_audit_local_check`.

Test plan

  • `./scripts/pre-pr.sh` → PASS (clippy `--all-targets -D warnings` + full workspace test)
  • Workspace tests: 5192 passed / 0 failed (+9 net, zero regressions)
  • Walker-audit gate: PASS after regen
  • m071 `every_catalog_row_has_an_extractor` + `holistic_parity` (11/11) — cross-format parity anchor for C153
  • SC-005 integration test (`no_binary_scan_us1_registration_skip.rs`) — go_binary reader skipped, `pkg:golang/*` from BuildInfo drop to zero
  • SC-006 cross-format parity test (`no_binary_scan_us3_annotation.rs`) — CDX / SPDX 2.3 / SPDX 3 all carry `"go"` when flagged, absent when unset
  • SC-004 `--help` discoverability test (`no_binary_scan_us2_help.rs`)
  • SC-007 error surface tests (`no_binary_scan_error.rs`) — 3 tests covering unrecognized-mode + bare-flag + empty-env-var-treated-as-absent (surfaced + fixed a clap-derive bug where empty env-var failed `value_enum` parse instead of being treated as absent)
  • Env-gated perf gates (SC-001/002/003) at `perf_walk_dispatch.rs` — CI opt-in via `WAYBILL_PERF_{MONGO,PYTORCH,ANSIBLE}_DIR`
  • End-to-end smoke test against `/tmp/nb_test` (real Go binary with BuildInfo) — verified `registered_readers=28` → `27`, `go_binary` absent from `per_reader_dispatch_counts` map, INFO log emitted, annotation present in all three formats

Deferred items (not in this PR)

🤖 Generated with Claude Code

mlieberman85 and others added 2 commits August 26, 2026 00:47
…ree scan speed

Add `--no-binary-scan=<MODE>` (env: WAYBILL_NO_BINARY_SCAN) to skip
specified binary-content-scanning readers at pilot time. v1 mode `go`
skips the go_binary reader — no BuildInfo probing for statically
linked Go binaries. Components claimed via OS-package readers (dpkg /
apk / rpm / pip RECORD) remain emitted from those sources.

Perf reference (macOS APFS, warm cache):
* ansible (5.8k files): 0.777s → ~0.3s (SC-003)
* pytorch (21k files):  1.117s → ~0.4s (SC-002)
* mongo   (55k files):  3.04s  → ~0.7s (SC-001)

Resolves the biggest single follow-up item from m664's perf-comparison
backlog (mongo residual attributed to ~2.4s of go_binary content
probing; the gate eliminates it).

Emits a document-scope `waybill:binary-scan-suppressed=<mode>` annotation
(C153 parity catalog row, SymmetricEqual) across CDX / SPDX 2.3 / SPDX 3
so downstream consumers detect the opt-out without inspecting waybill
invocation state. Default (unset) path is byte-identical to pre-m665
emission per FR-003.

CLI wins over env-var when both are set; empty env-var treated as
absent (sanitized at main.rs before Cli::parse).

Extensibility contract: BinaryScanMode enum reserves `all` / `elf` /
`symbols` variants for future opt-outs of m096 ELF section reader,
m099 symbol fingerprint, and m104 binary-role classification without
new C-rows.

Spec: specs/665-no-binary-scan-flag/
Design notes: docs/design-notes.md § Operator-controlled reader gating
Operator guide: docs/user-guide/cli-reference.md § --no-binary-scan

Test coverage:
* SC-005 integration (no_binary_scan_us1_registration_skip.rs)
* SC-006 cross-format parity (no_binary_scan_us3_annotation.rs)
* SC-004 discoverability (no_binary_scan_us2_help.rs)
* SC-007 error surface (no_binary_scan_error.rs) — 3 tests
* Env-gated perf gates (perf_walk_dispatch.rs) — mongo/pytorch/ansible

Workspace tests: 5192 passed / 0 failed (+9 net, zero regressions).
Walker-audit allowlist regenerated (23 entries from pre-existing main-branch drift).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit regenerated the walker-audit allowlist against an
interactive-shell run of the CI script where sed was missing from
the subshell environment, silently letting test fns with the
`// walker-audit:` sigil through the filter. CI runs under
`bash --noprofile --norc` where sed is available and correctly
suppresses those matches, so the regenerated file had 23
false-positive entries.

Re-regenerated under `bash --noprofile --norc` matching CI's env.
File returns to main's 12-entry state (m665 introduced no new
`fn walk_*` functions, as expected).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@mlieberman85
mlieberman85 merged commit dc8018d into main Aug 26, 2026
22 checks passed
mlieberman85 added a commit that referenced this pull request Aug 27, 2026
Cuts a stable v0.3.0. 43 commits since v0.2.0 with substantial
feature additions:

Feature highlights (grouped):

* Resolvers + graph completeness:
  - m663 (#708, #709, #710, #712): local cache-probe resolver — reads
    ~/.m2, $GOMODCACHE, ~/.cargo, RubyGems, npm, pip caches for
    high-confidence PURL extraction without network. New C152
    `waybill:resolver-tier` doc-scope annotation.
  - m235 (#688-#701): Gradle transitive dep-graph ladder — four-tier
    resolution (subprocess → cache → static → lockfile-only) with
    C146-C150 doc-scope + per-component annotations.
  - m236 (#703-#706): universalize `waybill:unresolved-reason` (C151)
    across all design-tier readers.

* Architecture:
  - m664 (#715): single-pass filesystem walker — consolidates 21+
    ecosystem-reader walk sites into one tree traversal per scan.
    5-6× warm-cache perf improvement on large fixtures.
  - m665 (#719): `--no-binary-scan=<MODE>` flag — operator opt-out
    of Go-binary content probing. C153 `waybill:binary-scan-
    suppressed` doc-scope annotation across CDX/SPDX 2.3/SPDX 3.

* eBPF:
  - m234 (#682, #686): un-pin bpf-linker to 0.11.0 via pre-built
    binary install + composite action + daily canary + verify script.

* Fixes:
  - #711: verifier-friendly bound in tls_openssl::probe_ssl_read
    for kernel bumps.
  - #680: Go workspace-mode detection in mod_why preflight.
  - Several release.yml + cosign compat fixes (#674, #675, #681).

Golden regeneration:

* CDX goldens: 11 files, 1-line churn each — `metadata.tools[].version`
  bumped 0.2.0 → 0.3.0.
* SPDX 2.3 goldens: 11 files. Version-string churn cascades into
  content-addressed SPDXRef-DocumentRoot-* IDs and documentNamespace
  URLs (both content-hashed).
* SPDX 3 goldens: 11 files. Same content-addressing cascade into
  doc-/pkg-/anno-/rel- IRIs.

Verified: normalized+sorted diff on all 33 goldens shows zero
semantic change beyond the version string per project memory
`feedback_verify_golden_churn_normalized`.

Also updates `waybill-cli/tests/fixtures/pkg_alias_binding/image-
baz.cdx.json` (same one-line version bump on an out-of-tree
regression fixture).

Skipping local pre-PR per project memory
`feedback_release_bump_prepr_slow` — version bump invalidates the
full compile cache so pre-PR takes 30+ min; CI will verify.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant