Skip to content

feat(agent): Evidence-Before-More-Mutation nudge (arm B, env-gated) - #7952

Open
esengine wants to merge 1 commit into
feat/verification-debt-telemetryfrom
feat/ebm-evidence-nudge
Open

feat(agent): Evidence-Before-More-Mutation nudge (arm B, env-gated)#7952
esengine wants to merge 1 commit into
feat/verification-debt-telemetryfrom
feat/ebm-evidence-nudge

Conversation

@esengine

@esengine esengine commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Stacked on #7947. First runtime policy of the intervention phase - Arm B of the causal experiment.

Trigger (calibrated, deliberately narrow)

verification_debt_open AND blind_mutations >= 3 AND not_fired_this_turn

Calibration from the two recorded baseline runs: healthy trajectories peak at blind=2 (edit, edit, script-check), so blind>=2 would have nudged 31/49 (63%) of currently-passing runs; blind>=3 touches 4/49 (8%) while still catching the one recorded failure (blind peak 5, nudged at round 10 of 15 with half the runtime left). The slow-reasoning-gap condition is deliberately excluded and reserved for Arm C, keeping B's causal question pure: "does earlier discriminating evidence have value?"

What

  • OutcomeTracker counts blind_mutations (mutations since the last discriminating observation; settles with debt).
  • applyEBM (internal/agent/ebm.go): stamps ebm_eligible on every arm - baseline carries the same shadow - and, only under REASONIX_EXPERIMENT_EBM=1, appends the nudge once per turn through the guard channel plus an evidence_nudge notice. Nudge copy explicitly permits finishing the minimum coherent multi-file change before checking.
  • e2ebench -policy ebm sets the env for suite task runs (arm switch for interleaved A/B).
  • Runtime records only the three per-round fields; e2ebench derives the full mechanism chain offline per run: ebm_eligible_round, ebm_fired_round, blind/debt at fire, ebm_rounds_to_check, ebm_ms_to_check, ebm_reasoning_to_check, ebm_check_within_1/2, plus debt_area and blind_peak. The Outcome-shadow report line gains EBM eligible K - fired F (compliance <=2 rounds P%, median rounds-to-check N) - EBM Compliance is the mechanism-health readout that separates "nudge ignored" from "early evidence useless".

Verification

  • New tests: blind counter lifecycle (accumulate across rounds, settle on discriminating check), eligibility-without-firing by default, fire-once-per-turn + reset with turn evidence + notice emission, e2ebench chain derivation + render.
  • go test across evidence, agent, e2ebench, trajectory, cli, boot, tool/builtin - all pass; gofmt/go vet/repolint clean.
  • Note: TestSnapshotterCapturesWorkspaceChanges (pre-existing, untouched by this diff) showed a timing flake under load during verification (300ms poll vs sleep boundaries); alternated branch runs confirmed it fails independently of this change.

Cache-impact: none - the nudge rides tool results in the turn tail exactly like the progress guard; the provider-visible prefix is byte-identical, and with the experiment env unset no prompt bytes change at all
Cache-guard: TestApplyEBMStampsEligibilityWithoutFiringByDefault pins the default-off path; progress-guard tests pin the shared guard channel
Documentation-impact: none - an env-gated experiment policy plus developer-tool report fields; nothing user-facing changes until the experiment graduates, at which point config + docs land together

@esengine
esengine requested a review from SivanCola as a code owner August 8, 2026 08:14
@github-actions github-actions Bot added the agent Core agent loop (internal/agent, internal/control) label Aug 8, 2026
The intervention phase's first runtime policy. Trigger: verification debt
open AND blind_mutations >= 3, at most once per turn. Calibrated against
two recorded baseline runs: healthy trajectories peak at two blind
mutations (edit, edit, check), so the threshold spares 92% of currently
passing runs, while the one recorded failure peaked at five and would
have been nudged at round 10 of 15 with half its runtime left.

The nudge asks for the cheapest discriminating evidence and explicitly
permits finishing the minimum coherent multi-file change first - the
goal is feedback, not test-after-every-edit. Enforcement is env-gated
(REASONIX_EXPERIMENT_EBM=1, e2ebench -policy ebm); eligibility is
stamped on every arm so baseline carries the same shadow, keeping the
A-vs-B comparison inside one dataset.

Runtime records only blind_mutations/ebm_eligible/ebm_fired per round;
e2ebench derives the whole mechanism chain offline: fire round, blind
and debt at fire, rounds/ms/reasoning-tokens to the first discriminating
check, compliance within 1 and 2 rounds, plus debt area and blind peak
per run - the readouts that separate "nudge ignored" from "early
evidence useless".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant