Skip to content

test: weekly mutation-score ratchet over decision kernels (Stryker, non-gating first) #1415

Description

@thymikee

Part of #1412 (Track A).

Context

When agents write both code and tests, mutation score is the only mechanical answer to "is this test load-bearing or decorative". Full-suite mutation is off the table; this targets an enumerated list of pure decision kernels where a surviving mutant means a silently wrong agent-facing decision.

Deliverables

  • Stryker (vitest runner) config scoped to exactly: src/kernel/errors.ts (retriability/hints), src/daemon/ref-frame.ts (ADR 0014 admission matrix), src/commands/interaction/runtime/settle.ts, src/utils/scroll-edge-state.ts, selector matching (src/selectors/).
  • Weekly scheduled workflow; report as job summary + artifact. Non-gating initially.
  • A per-module score baseline file with a ratchet check (scores may only rise), enabled once two consecutive weekly runs are stable.

Acceptance

  • Weekly run completes < 30 min on ubuntu runner.
  • Baseline file + ratchet script tested (a lowered score fails the check with the surviving-mutant list).

Out of scope

Any module that spawns subprocesses or waits real time; gating PRs on mutation score.

Review amendments (2026-07-27)

  • Define graduation criteria up front: after N stable weekly runs, mutation testing gates PRs for AFFECTED modules only (changed file in the kernel list → run its mutants on the PR); weekly stays the full sweep. Otherwise regressions are found only post-merge.
  • Store Stryker version + config content hash alongside each baseline; a score change caused by a tool/config change must be distinguishable from a test-strength change.

PR-wave review outcome (2026-08-01, PR #1441)

Approve-with-changes; real StrykerJS, exact five-kernel scope asserted at three points, both amendments implemented precisely (provenance-tagged baselines, drift vs regression classified, graduation mechanically enforced). Remaining items now part of this spec:

  • Strip or split the bundled obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 watcher; KEEP scripts/lib/lane-envelope.ts as the wave's shared envelope module.
  • Budget verification before trusting the schedule: kernel-errors measured 9m04s/183 mutants on a 2-core runner → the 1,277-mutant selectors shard projects to ~45–55 min, past the <30-min criterion and the 45-min shard timeout. One manual workflow_dispatch is required; re-shard or trim globs if the projection holds.
  • Maintainer decision required: mutation-affected.yml (PR-triggered, dormant-by-baseline) already spends real CI minutes pre-graduation. Approve the spend explicitly or gate the trigger on stableRuns >= 2.
  • Document the intentionally-missing tsconfig.stryker-absent.json trick with a link to the Stryker behavior it relies on.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions