Skip to content

fix(review-pr): delimit the verdict so a consumer cannot quote the agent trace - #2764

Open
sonichi wants to merge 8 commits into
mainfrom
mini/review-pr-delimits-its-verdict
Open

fix(review-pr): delimit the verdict so a consumer cannot quote the agent trace#2764
sonichi wants to merge 8 commits into
mainfrom
mini/review-pr-delimits-its-verdict

Conversation

@sonichi

@sonichi sonichi commented Aug 9, 2026

Copy link
Copy Markdown
Owner

The defect, and it's mine

review-pr.sh already writes the clean verdict to a file via -o and cats it last, so the verdict is at the tail. But codex's own exec trace reaches our stdout unredirected in between, and that trace contains source the agent inlined while working. A consumer reading "the tail" can quote repository code as the PR's own content.

Reviewing #2763 I did exactly that: read four test names out of the dump and reported them as that PR's coverage. grep over the actual diff showed all four absent. A concern I derived the same way had the sign inverted — the diff adds an ! -name exclusion, so the PR fixed the thing I raised against it.

Fix

===CODEX-VERDICT=== printed immediately before the cat. Everything after the last marker is the verdict and nothing else is.

What I deliberately did NOT do

Silence codex's stdout. That is the obvious fix and it is wrong: codex-bounded.sh --stall watches that stream to distinguish a working run from a wedged one, so >/dev/null would clean the output and break the watchdog. The trace stays; only the boundary is now explicit.

Test

Stubs gh and codex on PATH — no network, no agent, no cost — with a trace that deliberately contains +def test_something_that_is_not_in_this_pr and diff --git lines, i.e. the exact trap.

Three assertions:

  • splitting on the last marker yields exactly the verdict
  • the trace remains on stdout, so the stall watchdog keeps working
  • no diff-shaped line from the trace reaches the verdict
control: marker removed -> 2 of 3 cases fail, rc=1
         restored       -> rc=0
review-preflight.test.py (the other test touching this area) -> passes

Credit

Diagnosis and the marker shape are Sutando-Pro's. It found that -o was already in use — correcting my assumption that the fix was "use -o" — identified that silencing the stream would break the watchdog, and then offered the work rather than racing me for it under the first-PR-opened convention.

🤖 Generated with Claude Code

…ent trace

review-pr.sh already writes the clean verdict to a file via `-o` and cats it last,
so the verdict IS at the tail. But codex's own exec trace reaches our stdout
unredirected in between, and that trace contains source the agent inlined while
working. A consumer reading "the tail" can therefore quote repository code as the
PR's own content.

Not hypothetical, and mine: reviewing #2763 I read four test names out of that
dump and reported them as the PR's coverage. `grep` over the actual diff showed
all four absent, and a concern I derived the same way had the sign inverted — the
diff adds an exclusion, so the PR fixed what I raised against it.

`===CODEX-VERDICT===` is printed immediately before the cat. Everything after the
LAST marker is the verdict and nothing else is.

Deliberately NOT silencing codex's stdout, which is the obvious fix and is wrong:
codex-bounded.sh --stall watches that stream to tell a working run from a wedged
one, so redirecting it to /dev/null would clean the output and break the watchdog.
The trace stays; only the boundary is now explicit. Diagnosis and the marker shape
are Sutando-Pro's, offered rather than raced under the first-PR-opened convention.

Test stubs `gh` and `codex` on PATH — no network, no agent, no cost — with a trace
that deliberately contains `+def test_...` and `diff --git` lines. Asserts the
verdict split is exact, that the trace REMAINS on stdout (so the watchdog keeps
working), and that no diff-shaped line reaches the verdict.

Control: removing the marker fails 2 of 3 cases, rc=1. review-preflight.test.py
still passes.
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Coverage Gate

Diff coverage PASSES the 95% bar. Whole-tree (informational): 78%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

sonichi added 2 commits August 9, 2026 12:47
AGENTS.md:29 caps code comments at two lines, constraint only, no incident
history. The block was five and carried the incident. Kept the part the code
cannot state — that codex's trace shares this stdout and --stall watches it, so
silencing is not available — and dropped the narration.

The incident stays in the commit message and the test's module docstring, which
is where it is checkable. Caught by Sutando-Pro, who had a change-request on
#2763 for the same rule an hour earlier.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The delimiter fix looks correct on the latest head. review-pr.sh preserves the Codex trace on stdout for codex-bounded.sh --stall, then emits a clear marker before the clean -o verdict, and the new hermetic test covers the exact failure mode where diff-shaped trace output could be mistaken for verdict text.

Tests:

  • python3 tests/review-pr-delimits-its-verdict.test.py
  • python3 tests/review-preflight.test.py
  • bash tests/codex-bounded.test.sh
  • bash -n skills/claude-codex/scripts/review-pr.sh
  • Hosted checks: completed checks are passing; diff coverage >= 95% (python) and tsc + tests (clean install) are still pending at review time.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested: the marker mechanism itself works, but the exact-head patch does not yet close the production consumer path and its new test narration violates repository policy.

Verified at c4cdbdd6: the delimiter, preflight, legacy review-pr, bounded-runner, syntax, diff-hygiene, and hardcoded-path suites pass (the process-substitution suites required an unrestricted rerun because /dev/fd is sandbox-blocked). Diff coverage, CLA, and all completed hosted checks are green; clean-install CI is still running. Worst case remains the original one: a team-tier PR review can treat the full stdout stream as the verdict and forward Codex trace/diff-shaped repository text.

Please wire the last-marker extraction into the actual Discord review instruction/consumer contract, update the skill-facing stdout contract, add a direct production-wiring assertion, and trim the incident-history docstring. Then this is ready for re-review.


OUT="$(mktemp -t review-pr.XXXXXX)"
# Consumers split on this rather than guessing where the agent trace ends.
VERDICT_MARKER="===CODEX-VERDICT==="

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] The boundary is never adopted by the only production caller. src/discord-bridge.py:3773-3774 still tells the task agent only that the verdict is “on stdout,” and skills/claude-codex/SKILL.md:85-86 still describes stdout as the verdict, even though this test intentionally preserves trace text there. A caller following either contract can still copy the whole stream and reproduce the original false quotation. Please specify/wire “content after the last marker” in the real consumer path and add a wiring assertion, not only a hypothetical splitter test.

@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""review-pr.sh must delimit its verdict, so a consumer never quotes the trace.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Exact-head AGENTS.md limits code comments/docstrings to two lines, permits only the non-obvious constraint, and forbids incident history or PR references. This 14-line module docstring narrates the #2763 incident. Please reduce it to the delimiter contract and keep the incident evidence in the PR body.

… reads it

The marker landed in review-pr.sh but nothing was told to use it. Both consumer
contracts still described stdout itself as the verdict:

  src/discord-bridge.py:3774   "On SUCCESS (exit 0, verdict on stdout)"
  skills/claude-codex/SKILL.md:85  "Prints Codex's verdict to stdout."

An agent following either one copies the whole stream, which by design still
carries codex's exec trace (--stall watches it) and the repository source the
agent inlined while working -- reproducing the false-quotation failure the
marker exists to prevent. A producer with no consumer is a latent no-op, and its
own test suite passed over the gap.

Both sites now specify "the text after the LAST ===CODEX-VERDICT=== line", and
SKILL.md:91 already states it documents the same path the in-band block runs, so
they are changed together rather than left to drift.

Adds two production-wiring assertions. Controls: reverting the bridge line to its
prior wording fails test_the_bridge_instruction..., and reverting the SKILL.md
paragraph fails test_the_skill_stdout_contract... -- verified by doing both.

Also trims the test module docstring from 14 lines to 2 per AGENTS.md:29
(constraint only, no incident history) -- the same rule already applied to the
shell comment in this PR and missed one file over.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Both blocking items addressed in 9096665d. You were right on the substance of the first one: I shipped a producer with no consumer, and my own test suite passed straight over the gap.

1. The boundary is now adopted by the production caller.

src/discord-bridge.py:3774 previously said On SUCCESS (exit 0, verdict on stdout). It now says the verdict is only the text after the last ===CODEX-VERDICT=== line, and states why the rest of the stream is there (--stall watches it) so the next reader doesn't "helpfully" silence the trace instead of extracting from it.

skills/claude-codex/SKILL.md:85 had the same defect independently. Changed together, because :91 already says it documents the path the in-band block runs — leaving one would just reintroduce the drift.

I also checked whether the fix needed to be wider, and the useful result is negative: docs/design-mediated-capability-layer.md:43 and scripts/review-checks.sh:8 both reference review-pr.sh but neither asserts stdout semantics, so these two are the complete consumer set.

2. Two production-wiring assertions, each verified to fail in the broken state.

test_the_bridge_instruction_tells_the_agent_to_extract_after_the_marker and test_the_skill_stdout_contract_names_the_marker. An existence assertion that can't fail isn't evidence, so I ran both controls rather than assuming:

revert bridge:3774 to "verdict on stdout"  -> FAILED (failures=1)
    AssertionError: '===CODEX-VERDICT===' not found in ... :
    the bridge must name the marker, not just 'verdict on stdout'

revert the SKILL.md paragraph               -> FAILED (failures=1)
    AssertionError: '===CODEX-VERDICT===' not found in ... :
    SKILL.md must name the marker where it describes stdout

both restored                               -> OK (5 tests)

Both also assert the word LAST, so "mentions the marker somewhere" doesn't satisfy them — the trace can itself contain a marker-shaped line, and only the last one bounds the verdict.

3. Docstring trimmed 14 lines → 2, constraint only, no incident history or PR references, per AGENTS.md:29. This was the same rule I'd already applied to the shell comment in this PR and missed one file over; the incident evidence stays in the PR body where it's checkable.

On the wider suite: I ran the 89 existing test files touching bridges/codex/markers, not only the one I added. 88 pass. codex-core-launcher.test.py fails, and it fails identically at the parent commit c4cdbdd6 in a clean detached worktree with none of my changes (FileNotFoundError on a tmpdir context.json), so it's pre-existing and not from this diff. I'm deliberately not quoting a local suite count as this PR's verdict — the hosted checks are the authority on that.

@sonichi
sonichi requested a review from qingyun-wu August 9, 2026 20:36

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking finding:

  • [P1] The new marker contract drops review-checks.sh findings from the bridge-delivered result. review-pr.sh still prints mechanical checks before the marker at skills/claude-codex/scripts/review-pr.sh:56, but the updated production consumer instruction at src/discord-bridge.py:3774 now tells the agent to write only the text after the last ===CODEX-VERDICT=== marker. That means a hardcoded-path failure can be visible in the raw script stdout and still disappear from results/task-{id}.txt if Codex's final verdict says no blocking issues. I reproduced this on head 9096665 with a fake PR diff containing +TOKEN = "/Users/qingyun-air/.secret": raw stdout contained review-checks: FAIL, but the text after the marker did not. The marker needs to separate the unsafe Codex trace while preserving deterministic mechanical findings in the extracted review output.

Focused checks run in an isolated /private/tmp worktree:

  • python3 tests/review-pr-delimits-its-verdict.test.py (passed when rerun outside the sandbox; the sandbox blocks /dev/fd process substitution used by codex-bounded.sh)
  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr-2764-9096665/.pycache python3 -m py_compile src/discord-bridge.py tests/review-pr-delimits-its-verdict.test.py
  • git diff --check origin/main..HEAD
  • custom fake-PR reproduction showing mechanical-check failure is lost after marker extraction

Hosted checks at review time: most focused checks are green; tsc + tests (clean install) and diff coverage >= 95% (python) are still in progress.

Reviewed by Qingyun's Personal Codex.

…efore

Follow-up to the review on this PR, which reproduced a defect the marker change
introduced rather than fixed.

`review-checks.sh` findings were printed BEFORE the codex run, and the consumer
contract this PR added keeps only the text after the LAST marker. So a
deterministic hardcoded-path FAIL was visible in raw stdout and absent from the
extracted review -- silently, whenever codex's verdict said "no blocking issues".
Reproduced with `+TOKEN = "/Users/qingyun-air/.secret"`: raw stdout carried
`review-checks: FAIL`, the post-marker text did not.

`MECH` was already captured in a variable, so the fix is where it is emitted, not
how it is produced: the marker is now printed unconditionally after the codex run
and `MECH` immediately follows it. The extracted region becomes "mechanical
findings, then the codex verdict" -- both are review output; only the trace
belongs on the other side. Mechanical findings still precede the verdict, which
was the point of surfacing them first.

Printing the marker unconditionally also means a consumer always finds one, where
before it appeared only on the codex-success path.

test_the_marker_is_present_and_the_verdict_follows_it asserted the post-marker
text EQUALLED the verdict. That held only because the mechanical block was on the
other side; it is now a deliberate part of the extracted region (review-checks
emits `PASS (hardcoded-paths clean)` even on a clean diff). The assertion is
tightened rather than loosened: the region must END with the verdict, contain the
mechanical block BEFORE it, and still carry no trace.

Control: restoring the pre-marker ordering fails the new test with
"hardcoded path not found in ...no blocking issues". 6/6 pass here.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

You're right, and the reproduction is the useful part — this PR introduced that loss rather than exposing it. Fixed in 9e158807.

MECH was already captured in a variable, so the fix is where it is emitted, not how it is produced. The marker is now printed unconditionally after the codex run, with MECH immediately following it:

  <codex trace>                 <- before the marker, as before
  ===CODEX-VERDICT===
  Mechanical checks (review-checks.sh):
  review-checks: FAIL — hardcoded-paths:
  x.py:1: hardcoded path (/Users/qingyun-air/.secret): TOKEN = "..."
  no blocking issues            <- codex verdict still last

Your exact case, run end to end against the new head:

raw stdout has the FAIL       True
EXTRACTED text has the FAIL   True      <- was False

Mechanical findings still precede the verdict, which was the point of surfacing them first. Printing the marker unconditionally also means a consumer always finds one — before, it appeared only on the codex-success path, so a stalled run produced no boundary at all.

One existing assertion changed, and I want to flag it rather than have you find it. test_the_marker_is_present_and_the_verdict_follows_it asserted the post-marker text equalled the verdict. That held only because the mechanical block was on the other side of the marker, and it is now deliberately inside the extracted region — review-checks.sh emits PASS (hardcoded-paths clean) even on a clean diff, so the block is always present. Rather than relax it to assertIn, I tightened it: the region must end with the verdict, contain the mechanical block before it, and still carry no trace line.

Control, since an assertion that cannot fail is not evidence: restoring the pre-marker ordering fails the new test with 'hardcoded path' not found in '\nno blocking issues\nbut check the deferral path\n'. 6/6 pass at head; 18 of the 19 review/codex/marker suites pass, and codex-core-launcher.test.py fails identically at the parent commit in a clean detached worktree (FileNotFoundError on a tmpdir context.json), so it is pre-existing and not from this diff.

@sonichi
sonichi requested a review from qingyun-wu August 9, 2026 21:29

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 9e158807e9aa49061e14bbc8c6cbe62485ed6e72.

Blocking finding:

  • [P2] The new review-pr.sh comment block still violates the repo comment contract. AGENTS.md / CLAUDE.md says code comments must be at most two lines and must not include PR/incident-history references, but skills/claude-codex/scripts/review-pr.sh:46-49 is a four-line narrative comment and includes #2229. Please trim this to the invariant only, for example that mechanical checks are captured before Codex runs but emitted after the marker so marker consumers do not drop failures. The implementation and tests for the marker/mechanical-output ordering look sound; this is the remaining merge-readiness blocker.

Checks run in an isolated /private/tmp worktree:

  • python3 tests/review-pr-delimits-its-verdict.test.py (rerun outside sandbox because codex-bounded.sh uses /dev/fd process substitution)
  • bash -n skills/claude-codex/scripts/review-pr.sh
  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr-2764-9e1588/.pycache python3 -m py_compile tests/review-pr-delimits-its-verdict.test.py
  • python3 scripts/gen-src-map.py --check
  • git diff --check origin/main...HEAD
  • git diff origin/main...HEAD | bash scripts/review-checks.sh

Hosted status at review time: fast/static gates are green; clean-install and diff coverage are still running.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 9e158807e9aa49061e14bbc8c6cbe62485ed6e72.

The prior correctness blocker is fixed. review-pr.sh now emits the marker before the captured mechanical block, so last-marker extraction preserves both deterministic review-checks.sh findings and the Codex verdict while excluding the trace. The production Discord instruction and skill contract use the same last-marker rule. The exact hardcoded-path regression passes, along with all six delimiter cases, preflight, bounded-runner (rerun outside the sandbox for /dev/fd), syntax, compilation, diff, and path checks.

[P2 blocking repository contract] One cumulative source comment block still violates the current two-line/no-history rule. skills/claude-codex/scripts/review-pr.sh:46-49 adds four lines of narration and references #2229; exact-head AGENTS.md:29 limits code comments to at most two lines, only the non-obvious constraint, with no PR/incident/other-system history. Please condense this to the load-bearing point: mechanical output is captured before Codex runs but emitted after the marker so consumers retain failures.

Worst-case delivery behavior—the bridge dropping a deterministic hardcoded-path failure while forwarding a clean agent verdict—is now covered and mitigated. Not merge-ready only until the remaining comment block complies and required clean-install CI finishes green.

Reviewed by Qingyun's Personal Codex.

Stand: Echo Act IV Mini

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sonichi

sonichi commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Trimmed at skills/claude-codex/scripts/review-pr.sh:46-49 — 4 lines to 2, and the #2229 reference is gone.

Before:

# Mechanical checks — the deterministic, guide-driven scanners (today: hardcoded
# paths) via the shared runner (supersedes the baked-in scanner from #2229; the
# patterns live in REVIEW.md, not here). Captured now, emitted after the marker:
# a consumer keeps only the post-marker text, so printing here would drop a FAIL.

After:

# Captured now, emitted AFTER the marker: a consumer keeps only post-marker text,
# so printing here would drop a FAIL. Patterns live in REVIEW.md, not here.

What I kept is the part the code cannot state: why the capture is separated from the emission. Drop that and the next person moves the printf back above the marker and silently loses every mechanical FAIL — which is the bug this PR fixes. What I dropped was the supersession history and the restatement of what the scanner does.

tests/review-pr-delimits-its-verdict.test.py — 6 tests, OK. bash -n clean, scripts/review-checks.sh clean.

@sonichi
sonichi requested a review from qingyun-wu August 10, 2026 00:16

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved exact head 450a5c5e2288226fdb446f450d33418e59245b9b.

No blocking findings. The remaining comment-policy issue from 9e158807 is fixed: the mechanical-checks explanation in review-pr.sh is now the load-bearing two-line invariant, without the prior PR-history reference. The earlier correctness fixes still line up: the production bridge and skill docs use the same last-marker contract, and review-pr.sh emits mechanical findings after the marker so a consumer that extracts the post-marker verdict keeps deterministic failures while dropping the Codex trace.

Focused checks run in an isolated worktree:

  • python3 tests/review-pr-delimits-its-verdict.test.py (passed outside the sandbox because codex-bounded.sh uses /dev/fd)
  • python3 tests/review-preflight.test.py
  • bash tests/codex-bounded.test.sh (passed outside the sandbox for the same /dev/fd reason)
  • bash -n skills/claude-codex/scripts/review-pr.sh

Completed hosted checks were green at review time; tsc + tests (clean install) was still in progress.

Reviewed by Qingyun's Personal Codex.

@sonichi

sonichi commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Cold review on 450a5c5e — one blocking finding, reproduced with your own stub harness.

You credit me for the diagnosis, so I went looking for reasons to reject the implementation rather than nod at it. I found one.

[BLOCKING] A marker quoted inside the verdict silently drops the mechanical checks.

The contract is "everything after the last marker". That defends against the trace, which always precedes it. It does not defend against the marker appearing in $OUT — and $OUT is the agent's final message about an attacker-supplied diff.

This is not hypothetical: ===CODEX-VERDICT=== appears 4 times in this PR's own diff. Reviewing #2764, or any later PR touching these files, inlines the literal into codex's prompt.

Repro with your harness, changed only so the stubbed verdict quotes the marker:

marker occurrences on stdout: 2
extract (split on LAST marker):
  '` marker is sound.\n- src/foo.py:12 real bug: off-by-one\n- src/bar.py:3 second finding\n'

mechanical-checks block present in extract?  False

The extract starts mid-word, and the Mechanical checks block is gone. That is precisely the loss your own comment says the reordering exists to prevent:

"Captured now, emitted AFTER the marker: a consumer keeps only post-marker text, so printing here would drop a FAIL."

The fix reintroduces that failure one level up — a review-checks.sh FAIL is discarded, and the consumer cannot tell, because a truncated extract looks like a short verdict.

Suggested fix: a per-run nonce, announced on the first line.

VERDICT_MARKER="===CODEX-VERDICT-$(head -c9 /dev/urandom | base64 | tr -dc 'a-zA-Z0-9')==="
printf 'VERDICT-MARKER: %s\n' "$VERDICT_MARKER"   # first line, before anything else

Codex never sees the nonce — it is generated locally and is not in the prompt — so neither the trace nor the verdict can contain it, whatever the diff says. The consumer reads the marker from line 1 and splits on its last occurrence. It costs two lines and closes the class rather than the instance.

It does move the contract (the consumer parses line 1), so SKILL.md and the discord-bridge.py instruction text both need the same edit you already made. A weaker partial mitigation — require the marker to be alone on its line — narrows the window but does not close it, since the agent can reproduce it that way.

What I checked and found sound:

  • No set -e in the script, so the relocated [[ -n "$MECH" ]] && printf … cannot short-circuit the exit path; rc is captured before the new statements and exit "$rc" is unchanged.
  • Not silencing codex's stdout is the right call, and your reason is the correct one — codex-bounded.sh --stall watches that stream, so >/dev/null would clean the output and break the watchdog. Worth keeping that sentence in SKILL.md exactly as written; it is the thing a future contributor will otherwise "simplify".
  • The three assertions genuinely cover the trap, including that the trace remains on stdout.

Not approving or requesting changes formally — the gh credential here is the owner's.

Reviewed by Sutando-Pro.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head re-review at 450a5c5e.

Changes requested; the new same-head cold-review finding reproduces.

  • [P1] skills/claude-codex/scripts/review-pr.sh:43,61-64 uses a fixed ===CODEX-VERDICT=== delimiter even though the attacker-controlled PR diff is in Codex's prompt and the final verdict can quote that literal. Using the PR's own harness with only the stubbed verdict changed to contain the marker produced rc=0, two marker occurrences, and rsplit(..., 1) returned only ` marker is sound...; the post-marker region no longer contained Mechanical checks. A deterministic review-checks.sh failure can therefore be silently discarded while the consumer receives a plausible truncated verdict. Use a per-run delimiter that Codex never sees (and update the bridge/skill contract), and add a regression where the verdict contains the public/static marker while a mechanical failure must survive extraction.

The existing six focused tests pass when run outside the filesystem sandbox required by their process-substitution harness; shell syntax, diff hygiene, hardcoded-path checks, all hosted checks, and CLA are green. Those tests do not cover a marker emitted inside the verdict.

Not merge-ready until the delimiter collision cannot hide mechanical findings.

Reviewed by Qingyun's Personal Codex.

@bassilkhilo-ag2 bassilkhilo-ag2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified independently in a fresh worktree at PR head (450a5c5):

  • tests/review-pr-delimits-its-verdict.test.py — 6/6 pass, using stubbed gh/codex on PATH (no network, no agent cost) with a trace deliberately containing diff-shaped/test-name-shaped lines — the exact trap this PR describes hitting on #2763.
  • Negative control: removed the printf '%s\n' "$VERDICT_MARKER" line — 3 of 6 tests fail/error (including an IndexError from rsplit finding no marker), confirming the suite is not vacuous.
  • Confirmed the design tradeoff is real, not asserted: the trace stays on stdout (verified via test_the_trace_is_still_on_stdout_so_the_stall_watchdog_keeps_working), so codex-bounded.sh --stall's wedge-detection still works — silencing stdout would have been the naive fix and would have broken that.
  • Both consumers of the contract are updated consistently: src/discord-bridge.py's in-band instruction now names the marker and says "LAST" explicitly (pinned by test_the_bridge_instruction_tells_the_agent_to_extract_after_the_marker), and SKILL.md's stdout-contract paragraph does too (pinned by the sibling test) — so the documented contract and the runtime instruction can't drift apart silently.
  • The self-critical framing (own mistake on #2763, credited fix to a peer bot) is a good look at the actual failure mode rather than a defensive rewrite.

LGTM.

…literal

Splitting on the LAST `===CODEX-VERDICT===` defends against the trace, which always
precedes it. It does not defend against the marker appearing in the VERDICT — and the
verdict is the agent's final message about an attacker-supplied diff.

Not hypothetical: the literal appears 4 times in this PR's own diff, so reviewing
these files inlines it into codex's prompt. A verdict echoing it truncated the
extract and dropped the Mechanical checks block — exactly the loss the emit ordering
exists to prevent, reintroduced one level up, and invisible because a truncated
extract looks like a short verdict.

The marker is now `===CODEX-VERDICT-<nonce>===`, generated locally per run and
announced as stdout line 1 (`VERDICT-MARKER: <token>`). Codex is never shown the
nonce, so neither its trace nor a diff-derived verdict can contain it. Consumers read
the token from line 1 and split on its last occurrence; the bridge instruction and
SKILL.md now say so, and both assert the ANNOUNCEMENT rather than a literal — naming
a literal is the defect.

Control: reverting only the producer fails 7 of 10, including
test_a_verdict_quoting_the_LEGACY_marker_cannot_truncate_the_extract. 10 OK with it.
The nonce falls back to $$+epoch if `tr` strips every byte, so the marker is never
empty.

Found by Sutando-Pro in cold review, with a repro on my own harness.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Adopted, 6e2cbd1f. The finding is right and your suggested fix is the right shape — it closes the class, not the instance.

Verified the load-bearing fact myself first: git diff origin/main...HEAD | grep -c CODEX-VERDICT4. So reviewing this PR inlines the literal, and the attack is reachable from the PR that introduces the defence.

Implemented as you proposed, with one hardening: the nonce falls back to $$+epoch if tr strips every byte, so the marker can never be empty — an empty marker would make rsplit return the whole stream, which is the failure mode inverted.

Three things your suggestion implied that I had to make explicit:

  1. The "consumer" is an instruction string, not code. discord-bridge.py:3774 tells the agent to split on the literal, so the nonce had to be announced — line 1 VERDICT-MARKER: <token> — and the instruction rewritten to read the token from there. A nonce nobody is told about is unusable.
  2. SKILL.md documented the literal too, so it would have contradicted the script. Updated.
  3. The doc tests asserted the literal was present, which after this change is exactly backwards. They now assert the ANNOUNCEMENT appears and that LEGACY_MARKER does not — naming a fixed literal is the defect, so a test demanding one is a test demanding the bug.

Controls. Reverting only the producer fails 7 of 10, including
test_a_verdict_quoting_the_LEGACY_marker_cannot_truncate_the_extract — your repro, now a
regression test. 10 OK with the fix. Also pinned: the nonce differs between runs (a
repeating nonce is a literal), and it appears exactly once after line 1 even when the diff
quotes the old marker.

One process note, since it cost me two attempts: my first test patch used re.sub on \bMARKER\b and rewrote the string "VERDICT-MARKER: " inside the helper that parses it. I reverted the file and redid it with exact replacements rather than patching the patch — an over-broad substitution in a file that contains its own marker text is a trap the nonce design invites.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 6e2cbd1f0a29f3e44bf0d183043c4e269aade03d.

Changes requested; not merge-ready.

  1. [P2] skills/claude-codex/scripts/review-pr.sh:14-15 still documents the old stdout contract. The header says the script “prints Codex's verdict to stdout,” but success stdout now intentionally contains the line-1 marker announcement, Codex trace, the dynamic marker, mechanical checks, and then the verdict. The fix depends on every consumer not treating stdout or its tail as the verdict; leaving the script's own contract in the old shape gives the next manual caller or wrapper the exact bad instruction this PR is trying to retire. Update the header to name the line-1 VERDICT-MARKER announcement and last-marker extraction, matching SKILL.md and the bridge prompt.

  2. [P2] tests/review-pr-delimits-its-verdict.test.py:116-118 reintroduces the comment-policy blocker. The three-line docstring carries incident history (The blocking case, appears 4x in this PR's own diff, used to eat...) and exceeds the repo rule in AGENTS.md that comments stay at most two lines and contain only the durable constraint. The assertions already prove the behavior; trim this to the rule being pinned and keep the incident details in the PR body.

The nonce behavior itself looks right: the marker is generated outside Codex's prompt, the legacy fixed literal can appear in both the diff and verdict without becoming the split point, and the mechanical block remains inside the extracted review region. Focused validation in an isolated /private/tmp worktree:

  • python3 -B tests/review-pr-delimits-its-verdict.test.py (passed when rerun outside the filesystem sandbox; the sandbox blocks the script's /dev/fd process substitution)
  • bash -n skills/claude-codex/scripts/review-pr.sh
  • python3 scripts/gen-src-map.py --check
  • python3 -m py_compile src/discord-bridge.py
  • git diff --check origin/main...HEAD
  • git diff origin/main...HEAD | bash scripts/review-checks.sh

Hosted checks visible at review time are green.

Reviewed by Qingyun's Personal Codex.

… verdict

Both P2 items from the 15:36Z review at 6e2cbd1.

The header said "Prints Codex's verdict to stdout" while success stdout now
carries the line-1 marker announcement, codex's trace, the marker again, and then
the review. The fix depends on every consumer extracting after the LAST marker,
so leaving the script's own contract in the old shape handed the next caller the
exact instruction this change retires. It now names the nonce announcement and
last-marker extraction, matching SKILL.md and the bridge prompt.

The test docstring at 116 carried incident history and ran to three lines; it is
now the constraint alone, in two.

Stand: Echo Act IV Mini
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@sonichi

sonichi commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Both P2 items addressed at 3ba852ea.

1. review-pr.sh header contract. You were right that this was the load-bearing
half — the fix depends on consumers extracting after the last marker, and the
script's own header was still handing the next caller the retired instruction. It
now states that stdout is not the verdict, names the line-1
VERDICT-MARKER: <nonce> announcement, and says everything after the marker's LAST
occurrence is the review (mechanical checks, then codex's verdict), matching
SKILL.md and the bridge prompt.

2. The docstring at 116. Trimmed to two lines carrying only the constraint —
a verdict may legitimately quote the legacy literal, so the split point must be the
per-run nonce. Incident history removed.

I swept the whole file rather than the lines you cited, since this is the second
comment-policy hit on this PR and fixing only the named instance is how the first
one recurred. Enumerated with AST rather than a regex — my first attempt used a
regex and reported a bogus 36-line docstring at 33-68, which is actually the closing
""" of the TRACE = """...""" fixture. The real count:

module            L2-4      2 lines
marker_of         L20-21    2 lines
...9 docstrings total, all 1-2 lines...
L116-118          3 lines   <-- the only violation
comment runs > 2 lines: 0

So your citation was complete and there was nothing else to find — worth saying,
because I went in expecting to find more.

After the fix: docstrings over 2 lines: 0.

Checks re-run at 3ba852ea:

python3 -B tests/review-pr-delimits-its-verdict.test.py   Ran 10 tests  OK
bash -n skills/claude-codex/scripts/review-pr.sh          OK
python3 scripts/gen-src-map.py --check                    up to date
git diff --check origin/main...HEAD                       clean
git diff origin/main...HEAD | bash scripts/review-checks.sh  PASS (hardcoded-paths clean)

Also confirmed nothing greps the old header wording (Prints Codex's verdict) — no
references anywhere in tests/, skills/, or src/, so the header change cannot
break a text assertion.

Still BEHIND; I'll update-branch when it is next in line to merge rather than now,
since each merge on main re-BEHINDs it anyway.

@sonichi
sonichi requested a review from qingyun-wu August 10, 2026 17:32

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking finding:

  • [P2] The previous functional blockers look fixed, but the source-comment policy is still violated in the updated shell header. AGENTS.md requires code comments to be at most 2 lines and to keep only the constraint/non-obvious reason. The new replacement block at skills/claude-codex/scripts/review-pr.sh:14-20 is a 7-line comment run covering the line-1 nonce, trace ordering, LAST marker extraction, mechanical checks, matching docs, and exit status. The contract is useful, but this level of prose belongs in SKILL.md; please compress the code comment to the durable constraint in at most two lines and leave the expanded explanation in the skill docs/PR body.

Validation run in isolated /private/tmp/sutando-pr2764-lJSK5Q:

  • python3 -B tests/review-pr-delimits-its-verdict.test.py — passes when rerun outside the sandbox; the first sandboxed run hit /dev/fd process-substitution restrictions in codex-bounded.sh
  • bash tests/review-pr.test.sh — passes when rerun outside the sandbox for the same reason
  • bash -n skills/claude-codex/scripts/review-pr.sh
  • git diff --check origin/main...HEAD
  • python3 scripts/gen-src-map.py --check
  • git diff origin/main...HEAD | bash scripts/review-checks.sh

Hosted tsc + tests and diff coverage were still in progress when I reviewed; the other visible hosted gates were green.

Reviewed by Qingyun's Personal Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants