feat(agent): Evidence-Before-More-Mutation nudge (arm B, env-gated) - #7952
Open
esengine wants to merge 1 commit into
Open
feat(agent): Evidence-Before-More-Mutation nudge (arm B, env-gated)#7952esengine wants to merge 1 commit into
esengine wants to merge 1 commit into
Conversation
The intervention phase's first runtime policy. Trigger: verification debt open AND blind_mutations >= 3, at most once per turn. Calibrated against two recorded baseline runs: healthy trajectories peak at two blind mutations (edit, edit, check), so the threshold spares 92% of currently passing runs, while the one recorded failure peaked at five and would have been nudged at round 10 of 15 with half its runtime left. The nudge asks for the cheapest discriminating evidence and explicitly permits finishing the minimum coherent multi-file change first - the goal is feedback, not test-after-every-edit. Enforcement is env-gated (REASONIX_EXPERIMENT_EBM=1, e2ebench -policy ebm); eligibility is stamped on every arm so baseline carries the same shadow, keeping the A-vs-B comparison inside one dataset. Runtime records only blind_mutations/ebm_eligible/ebm_fired per round; e2ebench derives the whole mechanism chain offline: fire round, blind and debt at fire, rounds/ms/reasoning-tokens to the first discriminating check, compliance within 1 and 2 rounds, plus debt area and blind peak per run - the readouts that separate "nudge ignored" from "early evidence useless".
esengine
force-pushed
the
feat/ebm-evidence-nudge
branch
from
August 8, 2026 08:22
0f3386f to
dcbb41b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7947. First runtime policy of the intervention phase - Arm B of the causal experiment.
Trigger (calibrated, deliberately narrow)
verification_debt_open AND blind_mutations >= 3 AND not_fired_this_turnCalibration from the two recorded baseline runs: healthy trajectories peak at blind=2 (edit, edit, script-check), so blind>=2 would have nudged 31/49 (63%) of currently-passing runs; blind>=3 touches 4/49 (8%) while still catching the one recorded failure (blind peak 5, nudged at round 10 of 15 with half the runtime left). The slow-reasoning-gap condition is deliberately excluded and reserved for Arm C, keeping B's causal question pure: "does earlier discriminating evidence have value?"
What
OutcomeTrackercountsblind_mutations(mutations since the last discriminating observation; settles with debt).applyEBM(internal/agent/ebm.go): stampsebm_eligibleon every arm - baseline carries the same shadow - and, only underREASONIX_EXPERIMENT_EBM=1, appends the nudge once per turn through the guard channel plus anevidence_nudgenotice. Nudge copy explicitly permits finishing the minimum coherent multi-file change before checking.e2ebench -policy ebmsets the env for suite task runs (arm switch for interleaved A/B).ebm_eligible_round,ebm_fired_round, blind/debt at fire,ebm_rounds_to_check,ebm_ms_to_check,ebm_reasoning_to_check,ebm_check_within_1/2, plusdebt_areaandblind_peak. The Outcome-shadow report line gainsEBM eligible K - fired F (compliance <=2 rounds P%, median rounds-to-check N)- EBM Compliance is the mechanism-health readout that separates "nudge ignored" from "early evidence useless".Verification
go testacross evidence, agent, e2ebench, trajectory, cli, boot, tool/builtin - all pass;gofmt/go vet/repolintclean.TestSnapshotterCapturesWorkspaceChanges(pre-existing, untouched by this diff) showed a timing flake under load during verification (300ms poll vs sleep boundaries); alternated branch runs confirmed it fails independently of this change.Cache-impact: none - the nudge rides tool results in the turn tail exactly like the progress guard; the provider-visible prefix is byte-identical, and with the experiment env unset no prompt bytes change at all
Cache-guard: TestApplyEBMStampsEligibilityWithoutFiringByDefault pins the default-off path; progress-guard tests pin the shared guard channel
Documentation-impact: none - an env-gated experiment policy plus developer-tool report fields; nothing user-facing changes until the experiment graduates, at which point config + docs land together