fix(evidence): score a read by the question it asked, not just its path - #8407
Open
esengine wants to merge 1 commit into
Open
fix(evidence): score a read by the question it asked, not just its path#8407esengine wants to merge 1 commit into
esengine wants to merge 1 commit into
Conversation
The no-progress guard scored read novelty on the path alone, so the two most common investigation moves scored as repeats from the second round on: grepping one package for a second symbol, and paging through a long file. Measured on the real tool loop, a turn doing either was told it had "produced no new evidence" and was ordered to produce a final answer at round 7 — while every round had in fact returned information the turn did not have. A path already read can still answer a question never asked. Both scorers now key a read on the exact call behind the receipt as well as its paths, so a new grep pattern or the next window of a file counts once and only an identical call is a repeat. This also makes grep consistent with itself: a pathless grep was already scored by its arguments, so searching without a path scored better than searching one directory. The exploration run limit is untouched: past six look-only rounds a turn still stops earning novelty, which is a separate cliff.
github-actions
Bot
added
v2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #8404 so the smallest fix for the most frequent complaint can land
and be reverted on its own.
The no-progress guard scored read novelty on the path alone. Measured on the
real tool loop:
Nine of those ten rounds scored as repeats because the path had been seen. The
guard then told the model, untruthfully, that it had "produced no new evidence".
A path already read can still answer a question never asked. Both scorers now
key a read on the exact call behind the receipt as well as its paths: a new grep
pattern or the next window counts once, and only an identical call is a repeat.
It also makes grep consistent with itself — a pathless grep was already scored
by its arguments, so searching the whole tree scored better than searching one
directory.
Deliberately not in this PR: the exploration run limit still zeroes novelty
after six look-only rounds, so a read-only turn still reaches the stop tier by
arithmetic around round 12. That cliff, and the fixed 2/4/6 ladder it feeds, are
replaced in #8404.
Tests cover both scorers directly and both turn shapes through the real
executeBatchloop, plus the discriminator that must survive: re-reading onepath still escalates to the stop tier.
Cache-impact: none - evidence scoring only; no tool schema, prompt or provider request changes, so the system-prompt prefix stays byte-identical.
Cache-guard:
go test ./internal/boot/(provider_request golden) andinternal/agent/cachehit_e2e_test.go, both unchanged and green.Documentation-impact: none - the docs describe the guard's thresholds and pause behaviour, which this change does not alter; it only stops miscounting a distinct question as a repeat.