feat(m664): single-pass walker — 21+ reader migrations, ~5× perf

[mlieberman85]

Summary

• Milestone 664: single-pass filesystem walker + reader-registry. Replaces per-reader safe_walk (20+ tree traversals per scan) with ONE traversal + per-file/per-dir dispatch to interested readers. New crate module waybill-cli/src/scan_fs/walk_registry/.
• 21 readers + 3 originally-deferred migrated. Every production-path scan-tree walker now goes through the shared walker. Legacy pub fn read() retained under #[allow(dead_code)] per FR-004 coexistence.
• Contract C10 descend_into API extension lets readers opt into descending shared-walker-skipped dirs (target/, build/, dist/, etc.) with C10-scoped visibility so byte-identity is preserved for non-requesting readers.
• Perf validated against pre-m664 nightly. ansible 5.3×, pytorch 3.9×, mongo 5.2× wall-time improvement. See specs/664-single-pass-walker/perf-results.md + perf-comparison.md.

Perf validation (release-mode, warm-cache, macOS APFS, 2026-08-23)

Fixture	Files	Pre-m664 baseline	Post-m664 measured	Target	Improvement
ansible	5,793	4.10s	0.777s	≤ 1.2s (SC-001)	5.3×
pytorch	21,651	4.30s	1.117s	≤ 1.5s (SC-002)	3.9×
mongo	55,190	15.68s	3.04s	≤ 3.0s (SC-003)	5.2× (40ms over absolute, exceeds ≥5× spec claim)
SC-005 (10k synth)	10,000	—	34.6 µs/file p95	≤ 100 µs	beats 65%

Shared walker log confirms passes=1 on every scan. Walker cost per file: 4-12 µs. See specs/664-single-pass-walker/perf-results.md for the full write-up.

Head-to-head vs cdxgen / syft / trivy

Also compared against cdxgen 12.1.5 / syft 1.44.0 / trivy 0.71.1 on the same three fixtures. See specs/664-single-pass-walker/perf-comparison.md. Highlights:

• Walker cost per file is competitive — 4.4 µs/file on mongo, in line with syft/trivy.
• waybill leads on dep-graph coverage — mongo 571 edges vs syft 356 / trivy 172 / cdxgen 105.
• waybill still 1.9s slower than trivy on mongo — attributed to go_binary's 55k fs::metadata + read_binary probes (feature no other tool provides). Follow-up --no-binary-scan flag would close the gap.

Pre-m664 vs post-m664 byte-identity check

Ran pre-m664 nightly (v0.2.0-nightly.20260821) vs post-m664 HEAD on the three fixtures with identical flags. Two bugs surfaced and fixed same-session:

1. Symlink-to-file skip regression in walker.rs. Shared walker used entry.file_type().is_file() (does NOT follow symlinks); legacy safe_walk used Path::is_file() (follows). Dropped pytorch's docs/requirements.txt (symlink) + 22 sphinx/matplotlib/ipython deps. Fix: stat-follow symlink targets.
2. Composer + cocoapods missed dist/ subtree on mongo. Their legacy skip predicates were narrower than the shared walker default. Fix: added descend_into: [target, dist, build, out, coverage, bower_components] per contract C10.

Post-fix diff vs pre-m664: pytorch byte-identical; ansible +1 pkg:pypi/botocore (depth-10, coverage improvement from pilot depth 16 > legacy pip cap 6); mongo +1 pkg:cargo/gluesmith (same class). Remaining diffs are strict supersets — post-m664 finds MORE packages, never fewer.

Architecture (crate module waybill-cli/src/scan_fs/walk_registry/)

• walker.rs::SharedWalker — single-pass tree walker with canonicalize-keyed visited-set (m054 loop guard), m113 ExclusionSet, m114 permissive semantics.
• registry.rs::ReaderRegistry — dispatch table. Each ReaderRegistration carries a globset::GlobSet, optional on_file/on_dir, opaque per-scan state (Arc<dyn Any>), and new Option<globset::GlobSet> descend_into.
• dispatch.rs — insertion-order dispatch (C1) + catch_unwind per callback (C4 panic isolation).
• dir_index.rs::DirIndex — (dir → sorted-filenames) map for sibling lookup with zero extra syscalls (C6).
• walk_context.rs::SharedWalkerContext — reader handle with dir_index(), exclude_set(), push(), typed state::<T>().

10 contract clauses in specs/664-single-pass-walker/contracts/registry-api.md.

Migrations

Phase 1-2 (US1 pilot): haskell, scala, erlang, rpm_file, ipk_file — clean-shape criteria; collectively ~790ms savings on ansible baseline.

Phase 3-4 (US2 remaining): pip, cargo, gem, gradle, kotlin_dsl, npm, nuget, pants_jvm, pants_shell, pants_go, cocoapods, composer, dart, elixir, swift, vcpkg, conan, bazel, cmake, golang, yocto_layers. Patterns applied: Option<Vec<PathBuf>> precomputed-paths (T037 cargo / T047 pants_go / T058 golang / T059 yocto_layers), sibling-lookup on_dir (T041 gradle / T052 swift), marker-detect (T045 pants_jvm / T053 vcpkg / T054 conan / T055 bazel / T056 cmake), ancestor-path filter for legacy-only skips (T048 cocoapods / T051 elixir / T058 golang).

Phase 5-6 (US3 close-out): audit + escape-hatch annotations, walker-audit rationale companion doc, CI diagnostic pointer, SC-005 microbenchmark, docs updates.

Originally deferred, all resolved 2026-08-23:

• T029 yocto/recipe via Option<Vec<PathBuf>> precomputed-paths.
• T039 maven via C10 descend_into: [target].
• T057 go_binary via two-phase pilot pattern + C10 descend_into: [build, dist, out, coverage, venv].

Test plan

• ./scripts/pre-pr.sh green (clippy + workspace test)
• Workspace test 5183 passed / 0 failed
• Walker-audit CI-shape reproduction empty diff
• SC-001 (ansible ≤ 1.2s warm): measured 0.777s ✓
• SC-002 (pytorch ≤ 1.5s warm): measured 1.117s ✓
• SC-003 (mongo ≤ 3.0s warm): measured 3.04s (40ms over absolute; ≥5× improvement met at 5.2×)
• SC-005 (10k-file p95 dispatch ≤ 100µs): measured 34.6µs ✓
• Pre-m664 vs post-m664 diff: strict superset on ansible/mongo; pytorch byte-identical
• CI green (waiting on run)

Follow-up backlog (post-m664, out of scope for this PR)

Documented in specs/664-single-pass-walker/perf-comparison.md §Follow-up backlog:

1. --no-binary-scan flag (biggest single win — mongo would drop from 3.04s to ~640ms; ~1 day).
2. Extension-based cheap-reject in go_binary (half-day; small win).
3. Startup-cost audit (~200-300ms fixed cost on small fixtures; ~1 day investigation).
4. Parallel reader dispatch (FR-012 explicitly deferred by m664 so the single-pass baseline could be measured cleanly first).
5. --discovery-only mode (syft-style shallow scan; ~2 days).

🤖 Generated with Claude Code