Testing audit before v4.0.0

[joshfactorial]

A test-adequacy audit before v4.0.0 ships, after #710 removes the rneat alias. It covers item 2 of #803 ("Test adequacy"), split out so the release does not wait on the rest of that review.

The standard is CLAUDE.md's vetting section: tests assert content, survive a mutation check, and cover the path that can break.

Scope

1. Measure the harness coverage. The CLAUDE.md example under rule 6 ("scripts/delta/tests/ covers 2 of sv_pipeline.sbatch's 14 functions; score_caller and check_denominator … are untested") is out of date. The script now has 29 functions, scripts/delta/tests/ has 18 test files, and test_scorer_calibration.sh references score_caller. Record which functions are tested and which decisions those tests check, and update CLAUDE.md with the result.
2. #[ignore] tests pass. Run cargo test --workspace -- --ignored. Each must pass; a known-failing target does not belong there (gate2_realigned_dup is red on develop and main — the gate's denominator, not the duplication #582).
3. Mutation spot checks on the tests guarding numbers that get quoted: denominator and recall checks, model fitting (gen-seq-error-model, gen-mut-model), and truth/reads agreement. Confirm each mutant was applied.
4. Existence-only assertions. Find tests whose only check is that a file exists, a count is non-zero, is_ok(), or exit 0, and give them a content assertion.
5. Log assertions read the right channel. simplelog's TerminalMode::Mixed sends INFO and WARN to stdout and only ERROR to stderr, so asserting a warning's presence or absence on stderr alone proves nothing (found in Stop warning that a breakend has no END/SVLEN (#497) #818). A grep of eidolon/tests/ on 2026-10-05 found no such test; check new ones.
6. Python without CI. Settled 2026-10-06 in Describe sbs96_compare.py as it is: stdlib-only, kept untested #820: scripts/delta/sbs96_compare.py stays as is, untested, with no current caller. Tests get added if it is needed again; otherwise it goes when the validation scripts are archived. Not part of this audit.

Output

Small fixes land directly. Larger findings go in their own tickets. The measured state replaces the stale example in CLAUDE.md, and new case histories go in docs/claude_engineering_audit.md.

[joshfactorial]

Steps 1–2 done (#822)

• #[ignore] tests: all 8 run and pass in release-gates on every PR into develop. Each fails, rather than skips, when its tool is missing. No change needed beyond the header count.
• Harness: 18 suites, all passing; 232 mutations, all caught. CI runs both modes.
• sv_pipeline.sbatch: 25 of 29 functions exercised.
• Fixed in Test check_query_denominator; record measured harness coverage (#819) #822: check_query_denominator was never extracted into its suite, so the suite called it as an undefined command and nothing noticed. It now has tests and three mutations.

Untested, not fixed:

• score_caller (111 lines; assembles the recall figures in ACCESS §3.5–3.7, including the BND/CNV exclusion from the aggregate query)
• build_bnd_spans (three-way exit status: spans / legitimately none / defect)
• convert_manta_inversions (checks Manta's conversion rather than trusting its exit code)
• index_and_align (index-rebuild guard)

Next: step 3 (existence-only assertions).

[joshfactorial]

Step 3 done (#823)

47 tests flagged by scan → 24 OK (assertions in helpers), 22 weak and fixed, 1 needing a decision. Each fix has a hand-derived expected value and a mutation check. Two would never fail for the reason their name gives: test_all_n_contig_is_skipped never reached the N path, and test_runner_min_reads_zero_skips_filter could not see the filter.

Bug filed: #824, filter-reads output naming. Plain .fastq/.vcf panics; dotted directories break.

Open decisions:

1. gen-mut-model counts a SNP before checking its REF, so SNPs skipped for a REF mismatch (and at contig edges) still count toward mutation_rate and snp_freq (runner.rs:110 vs :127; measured: one good plus one mismatched SNP gives rate 2/13114).
2. gen-mut-model does not check indel REFs against the reference. The old test_runner_with_indels fixture had two wrong ones and nothing noticed.
3. A BED naming no reference contig fails as "Trinuc counts are empty. Unknown error". test_runner_bed_unknown_contig_succeeds_or_errors_cleanly accepts any outcome.
4. From step 2: score_caller, build_bnd_spans, convert_manta_inversions and index_and_align are untested. Do they hold v4.0.0, or go to a ticket?

Next: step 4 (mutation spot checks on the tests behind quoted numbers), then step 5.

[joshfactorial]

Decisions settled 2026-10-07:

• REF-mismatch policy and indel REF check: discard with a warning, refuse above 1%; indels checked the same as SNPs. BED error now names both contig lists. All in gen-mut-model: refuse REF mismatches as NEAT2 did, with a bcftools fix in the error #826, after comparing against NEAT2.
• Untested harness functions → Harness tests for score_caller, build_bnd_spans, convert_manta_inversions, index_and_align #825 (v4.1.0).
• filter-reads naming bug → filter-reads crashes on plain .fastq/.vcf and mis-handles dotted directories #824 (v4.0.0).