Description
A meaningful minority of the test suite asserts on the exact wording of human-readable text — error sentences, log lines, progress messages, and agent-facing notes — rather than on the behavior that text accompanies. This proposal establishes a written convention for how messages may be asserted, converts the assertions that pin wording without cause, and introduces an acceptance check that proves each conversion did not silently discard coverage.
Motivation
Pinning exact prose makes the suite fail on changes that alter no behavior: a typo fix, a punctuation change, or a clearer phrasing turns the build red. The costs compound in three ways.
First, it discourages improving the text. Every copy edit becomes a multi-file change, so wording that should be refined for the consuming model is left alone because touching it is expensive.
Second, it teaches contributors — human and AI — that pinning wording is the house style. Because there is no written convention, the existing precedent is the only guidance available, so each new message arrives with a new exact-text assertion, and the problem grows on its own.
Third, and most seriously, several of these assertions have already stopped working without anyone noticing. A negative assertion on prose becomes vacuously true once the wording changes, and then passes forever with no signal. The audit behind this proposal found assertions sitting inside provably-empty loops, a test whose mock supplies the very string its own regex checks, and integration tests that gate a skip on the wording of a production note — so a reword silently changes which code path the test takes and reports the outcome as a data problem.
A prototype also showed that the conversions themselves are not automatically safe. Of four proposed loosenings validated empirically, two turned out to drop real coverage: one lost the ability to detect a completion message copy-pasted from an unrelated tool, and one left a case-sensitivity hole in an invariant that forbids claiming success after a partial failure. Both passed their tests and looked correct on review. That is the strongest argument for treating "the loosened test still passes" as no evidence at all.
Current State
Roughly ninety to a hundred and ten assertions across the suite are coupled to wording that no consumer depends on. They cluster into a modest number of recurring shapes: progress and log text asserted by equality; a single error sentence duplicated verbatim across a route-test module nineteen times; error message phrases used as the only discriminator between distinct failure modes that production code raises with identical text; positional indexing into agent-facing note lists, which couples tests to the order notes happen to be appended; and negative assertions with no positive counterpart.
The suite also contains several places where pinning the wording is legitimate and works well — tool descriptions guarded by relational checks and length budgets rather than sentence equality, expectations built from imported constants, and keyword assertions that survive rewording while retaining teeth. These are the models to generalize from, but nothing records that they are the intended pattern.
No written guidance on the subject exists. The testing rule that would naturally host it is additionally scoped to a single subtree, so it does not apply to the API, resource, or top-level test modules where most of the affected assertions live.
A significant secondary finding: an audit conducted by pattern-matching over the suite over-reported by roughly a factor of two. Several flagged locations were already compliant, and several cited locations did not contain what was claimed. Any work here must verify each site by reading it, not by searching for it.
Proposed Changes
- Establish a written convention for asserting on messages, binding across the entire test suite rather than one subtree, that states when exact wording may be pinned and what to assert instead when it may not.
- Make failure modes distinguishable structurally rather than textually: distinguish error conditions by type or machine-readable code where a message phrase is currently the only discriminator, and observe progress and logging through call structure and record fields rather than rendered prose.
- Give the deliberately-tuned agent-facing message literals a single home, while keeping tests explicit about whether they verify that a message was delivered to the right place under the right condition, or verify its content — an assertion derived from the same constant it checks proves nothing about the wording.
- Guarantee that negative and order-dependent assertions cannot pass vacuously: pair every absence check with evidence that the text was produced at all, and assert membership rather than position unless order is itself the contract.
- Prohibit conditioning test control flow — skips, branches, expected-value selection — on production or third-party prose, since a reword must never change which path a test takes.
- Adopt an acceptance check for this class of refactor: before a wording assertion is replaced, demonstrate against a small set of deliberate defects that the replacement still detects everything the original detected, and record any deliberate reduction in writing.
Expected Benefits
- Message wording becomes editable without a test change, so agent-facing text can be tuned as the product it is rather than treated as frozen.
- The recurring precedent that produces new exact-text assertions is replaced by explicit guidance, so the problem stops growing with each added message.
- Assertions that have silently stopped testing anything are found and repaired, and the shapes that allow silent disarming become prohibited rather than merely discouraged.
- Coverage increases in places where identical error text currently makes several distinct failure modes indistinguishable, so tests begin discriminating branches they only appeared to cover.
- Failing tests become more diagnostic: an outage reports as an outage rather than as an unrelated assertion failure, and a genuine regression stops being reported as a skip.
- The refactor itself becomes reviewable and auditable, since each conversion carries durable evidence of what the assertion protects rather than relying on a reviewer's judgement at one point in time.
Description
A meaningful minority of the test suite asserts on the exact wording of human-readable text — error sentences, log lines, progress messages, and agent-facing notes — rather than on the behavior that text accompanies. This proposal establishes a written convention for how messages may be asserted, converts the assertions that pin wording without cause, and introduces an acceptance check that proves each conversion did not silently discard coverage.
Motivation
Pinning exact prose makes the suite fail on changes that alter no behavior: a typo fix, a punctuation change, or a clearer phrasing turns the build red. The costs compound in three ways.
First, it discourages improving the text. Every copy edit becomes a multi-file change, so wording that should be refined for the consuming model is left alone because touching it is expensive.
Second, it teaches contributors — human and AI — that pinning wording is the house style. Because there is no written convention, the existing precedent is the only guidance available, so each new message arrives with a new exact-text assertion, and the problem grows on its own.
Third, and most seriously, several of these assertions have already stopped working without anyone noticing. A negative assertion on prose becomes vacuously true once the wording changes, and then passes forever with no signal. The audit behind this proposal found assertions sitting inside provably-empty loops, a test whose mock supplies the very string its own regex checks, and integration tests that gate a skip on the wording of a production note — so a reword silently changes which code path the test takes and reports the outcome as a data problem.
A prototype also showed that the conversions themselves are not automatically safe. Of four proposed loosenings validated empirically, two turned out to drop real coverage: one lost the ability to detect a completion message copy-pasted from an unrelated tool, and one left a case-sensitivity hole in an invariant that forbids claiming success after a partial failure. Both passed their tests and looked correct on review. That is the strongest argument for treating "the loosened test still passes" as no evidence at all.
Current State
Roughly ninety to a hundred and ten assertions across the suite are coupled to wording that no consumer depends on. They cluster into a modest number of recurring shapes: progress and log text asserted by equality; a single error sentence duplicated verbatim across a route-test module nineteen times; error message phrases used as the only discriminator between distinct failure modes that production code raises with identical text; positional indexing into agent-facing note lists, which couples tests to the order notes happen to be appended; and negative assertions with no positive counterpart.
The suite also contains several places where pinning the wording is legitimate and works well — tool descriptions guarded by relational checks and length budgets rather than sentence equality, expectations built from imported constants, and keyword assertions that survive rewording while retaining teeth. These are the models to generalize from, but nothing records that they are the intended pattern.
No written guidance on the subject exists. The testing rule that would naturally host it is additionally scoped to a single subtree, so it does not apply to the API, resource, or top-level test modules where most of the affected assertions live.
A significant secondary finding: an audit conducted by pattern-matching over the suite over-reported by roughly a factor of two. Several flagged locations were already compliant, and several cited locations did not contain what was claimed. Any work here must verify each site by reading it, not by searching for it.
Proposed Changes
Expected Benefits