feat(ci): test the Codex support window at both ends, and report upstream drift (#408) - #550
Merged
Merged
Conversation
…ream drift (#408) Three of #408's four acceptance criteria, all credential-free. AC-2 is not here and says so. AC-1 - the manifest advertised `Codex >= 0.143.0; live-verified on 0.144.1` while CI installed exactly ONE version. The lower bound of the supported range rested on a single manual check from July; a range nothing exercises at its edge is a claim, not a guarantee. The required real-host lane now runs at both ends of the window, 0.143.0 and 0.145.0. Taken via AC-1's FIRST arm rather than its second: narrowing the claim to one tested version would have told working 0.143 installs they are unsupported. Both npm versions were verified installable before matrixing on them. The contract test no longer pins a literal. It DERIVES the requirement from the workflow: the manifest description must name every version the required lane installs, so bumping the matrix without amending the claim fails, and so does the reverse. It used to assert the string "0.144.1" - guarding the sentence rather than the fact. AC-3 - the real-host check verified that the installed hook manifest matched the SOURCE manifest. The install is a fresh copy of that same source, so for anything source declares the two agree BY CONSTRUCTION. Measured against a real host: a hook added to source with no script behind it left that check passing. It can catch the install dropping a file, which is its stated purpose, and nothing else. So every plugin-root-relative script the INSTALLED manifest points at must now exist in the install. A hook the host cannot execute is indistinguishable from one that allowed the operation, which is exactly the failure this lane exists to make visible. Verified against codex-cli 0.145.0 on this machine: 9/9 checks pass clean, and a declared-but-absent hook is KILLED with the missing path named. AC-4 - the required lane PINS its versions, so by construction it can never report that upstream moved. An advisory lane now installs @latest and runs the identical check. Awaited by ci-passed so the run settles, never in required_results: an upstream release must not be able to block a merge. It absorbs failure at STEP level rather than carrying a job-level continue-on-error, which would still render the check run as failure (#381's lesson). Already earning it: npm latest is 0.146.0 while the required lane pins 0.145.0 and the manifest said live-verified on 0.144.1. Nothing in CI reported that. AC-2 remains open and needs a decision, not more work: proving a hook FIRES needs a turn, a turn needs a model, and a provider credential cannot gate fork PRs. That boundary was already documented in this lane; this change does not move it. ca-codex 0.3.1 -> 0.3.2: the manifest description is shipped payload. Claude-Session: https://claude.ai/code/session_01WJgVfZw7J81PB7mwpHyUxx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three of #408's four acceptance criteria, all credential-free. AC-2 is not here, and says so.
AC-1 — the supported range had an unexercised edge
The manifest advertised
Codex >= 0.143.0; live-verified on 0.144.1while CI installed exactly one version. The lower bound of the supported range rested on a single manual check from July. A range nothing exercises at its edge is a claim, not a guarantee.The required real-host lane now runs at both ends — 0.143.0 and 0.145.0. Both were verified installable from npm before matrixing on them.
Taken via AC-1's first arm rather than its second: narrowing the claim to one tested version would have told working 0.143 installs they are unsupported.
The contract no longer guards the sentence
It used to assert the literal string
"0.144.1"— which made the manifest's support claim a statement about a past afternoon. It now derives the requirement from the workflow: the description must name every version the required lane installs. Bumping the matrix without amending the claim fails, and so does the reverse.AC-3 — a check that agreed with itself by construction
The real-host check verified that the installed hook manifest matched the source manifest. But the install is a fresh copy of that same source, so for anything source declares the two agree by construction.
Measured against a real host — I added a hook to source with no script behind it:
It can catch the install dropping a file, which is its stated purpose, and nothing else.
So every plugin-root-relative script the installed manifest points at must now exist in the install. A hook the host cannot execute is indistinguishable from one that allowed the operation — precisely the failure this lane exists to make visible.
Verified against
codex-cli 0.145.0on this machine:AC-4 — the pinned lane cannot report drift
By construction: it pins. An advisory lane now installs
@openai/codex@latestand runs the identical check.Awaited by
ci-passedso the run settles, never inrequired_results— an upstream release must not be able to block a merge. It absorbs failure at step level rather than carrying a job-levelcontinue-on-error, which would still render the check run as FAILURE (#381's lesson, and the Pi canary's shape).Already earning it: npm
latestis 0.146.0 while the required lane pins 0.145.0 and the manifest said live-verified on 0.144.1. Nothing in CI reported that.AC-2 stays open, and needs a decision rather than more work
Proving a hook fires needs a turn, a turn needs a model, and a provider credential cannot gate fork PRs. That boundary was already documented in this lane when the credential-free half shipped; this change does not move it.
Verification
test_codex_adapter82,test_ci_impact54,test_public_codex_docs3,test_validate_codex_plugin6,test_license_consistency23 — all pass. Real-host check exercised locally against a live Codex, both clean and mutated.ca-codex0.3.1 → 0.3.2 — the manifest description is shipped payload.