Skip to content

P0: auth-gate head-scan + factual doc corrections #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alanshurafa merged 2 commits into from
Jul 5, 2026
Merged

P0: auth-gate head-scan + factual doc corrections #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
640bae9
fix: catch long auth-error pages in agent output gate
alanshurafa Jul 5, 2026
fbda24b
docs: correct stale bouncer/model/CI/marker claims
alanshurafa Jul 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .planning/codebase/CONCERNS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Concerns

> **Historical snapshot (pre-v1.0).** The "no tests / no CI" claim below is stale: the repo now has a hermetic simulation suite (`tests/run-all.sh`, 25+ sims) running on 3-OS CI.

## 1. No Automated Regression Net

- The repository has no tracked automated tests and no CI.
Expand Down
2 changes: 2 additions & 0 deletions .planning/codebase/STACK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Stack

> **Historical snapshot (pre-v1.0).** The "no CI configuration" claim below is stale: the repo now has a hermetic simulation suite (`tests/run-all.sh`, 25+ sims) running on 3-OS CI.

## Snapshot

- Project type: tooling repo for cross-AI document refinement, not a deployable app or service.
Expand Down
6 changes: 3 additions & 3 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Co-Evolution is a tooling repo for structured iterative refinement between AI ag

## Snapshot
- Project type: tooling repo for cross-AI document refinement, not a deployable app or service.
- Primary executable: `agent-bouncer/agent-bouncer.sh`.
- Legacy runner (still used by tests/experiments): `agent-bouncer/agent-bouncer.sh`.
- Primary orchestrator spec: `skills/dev-review/SKILL.md`.
- Supporting assets: markdown templates in `agent-bouncer/templates/` and `skills/dev-review/templates/`, plus JSON schema in `skills/dev-review/schemas/review-verdict.json`.
- Tracked source is small: one shell entrypoint, one large skill spec, supporting docs, prompts, and schema files.
Expand All @@ -68,7 +68,7 @@ Co-Evolution is a tooling repo for structured iterative refinement between AI ag
## Runtime Dependencies
- `agent-bouncer/agent-bouncer.sh` assumes a POSIX shell plus standard utilities such as `date`, `head`, `tr`, `cp`, `mv`, `rm`, `wc`, `awk`, `tee`, `mkdir`, and `cat`.
- The bouncer depends on authenticated AI CLIs: `claude` and `codex`.
- The Claude adapter is hard-coded to `claude -p --output-format text --model claude-opus-4-6 --tools ""`.
- The Claude adapter defaults to model `claude-opus-4-6`, overridable via the `CLAUDE_MODEL` env var or the `--claude-model` flag (see `lib/co-evolution.sh`); it is not hard-coded.
- The Codex adapter is hard-coded to `codex exec --full-auto --skip-git-repo-check`.
- `skills/dev-review/SKILL.md` assumes Claude Code tooling, `git`, and optionally `gh` for PR creation.
## Build And Packaging
Expand Down Expand Up @@ -130,7 +130,7 @@ Co-Evolution is a tooling repo for structured iterative refinement between AI ag

## System Shape
- The repository contains two related but separate delivery surfaces:
- `agent-bouncer/agent-bouncer.sh` is the executable runtime for bouncing a document between agents.
- `agent-bouncer/agent-bouncer.sh` is a legacy runner (still used by tests/experiments) for bouncing a document between agents.
- `skills/dev-review/SKILL.md` is a declarative Claude Code workflow for compose -> bounce -> execute -> verify.
- Shared behavior is expressed through prompt templates rather than through a shared library module.
## Core Bouncer Flow
Expand Down
11 changes: 9 additions & 2 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ bash ./co-evolve-bouncer.sh --vanilla --chain "Should we ship this migration?"

### Agent Bouncer (`agent-bouncer/`)

Legacy standalone script that bounces any markdown document between two agents.
Legacy runner (still used by tests/experiments) that bounces any markdown
document between two agents.

```bash
bash agent-bouncer/agent-bouncer.sh <document.md> [max-bounces] [reviewer-agent] [composer-agent]
Expand Down Expand Up @@ -103,5 +104,11 @@ Co-evolution tools are integrated into GSD workflows:
## Conventions

- Plan content is embedded inline in prompts, never passed as a canonical file path
- Markers auto-expire after 2 passes to guarantee convergence
- Bounce stops after MAX_BOUNCES=2 passes; unresolved markers are reported in the output, not auto-expired
- Agent-bouncer overwrites the input file in place; orchestrators should back up first

## Status

- v1.4 npm/MCP publish pending — a human gate, not yet shipped.
- v1.5 Phase 6 partial — degrade-path dogfood only; the rest is unbuilt.
- Current default Claude model is `claude-opus-4-6` (overridable via `CLAUDE_MODEL` / `--claude-model`).
10 changes: 6 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,10 @@ bash ./co-evolve-bouncer.sh --vanilla --chain "Should we ship this migration?"

### [Agent Bouncer](agent-bouncer/)

A standalone bash script that bounces any markdown document between two agents.
Agent-agnostic: ships with Claude and Codex adapters, and new agents can be
added by writing one function. Runs from any terminal.
A legacy runner (still used by tests/experiments) that bounces any markdown
document between two agents. Agent-agnostic: ships with Claude and Codex
adapters, and new agents can be added by writing one function. Runs from any
terminal.

```bash
# Pass any document you want refined
Expand Down Expand Up @@ -199,7 +200,8 @@ Early development. The Co-Evolution skill, Co-Evolve Bouncer, Agent Bouncer,
Dev-Review skill, and standalone Codex runtime are functional and run
cross-platform on macOS, Linux, and Windows (Git Bash or WSL) — shell scripts
are pinned to LF endings and Windows/WSL path arguments are normalized
automatically. Next steps:
automatically. The 3-OS CI suite validates the hermetic (stubbed-CLI)
simulations, not live-LLM runs. Next steps:

- Additional agent adapters (Gemini CLI, Ollama, direct API calls)
- Standalone bounce protocol spec
Expand Down
73 changes: 62 additions & 11 deletions lib/co-evolution.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -576,6 +576,57 @@ file_contains_auth_failure() {
grep -qiE 'Failed to authenticate|authentication_error|Not authenticated|Not logged in|Unauthorized|login required|Please run .* login|Please run /login' "$file_path"
}

# A-2: strict, anchored auth-banner detector for the agent OUTPUT path. The
# broad file_contains_auth_failure above is a loose substring scan (it even
# matches a bare "Unauthorized"), which is right for stderr/empty-output paths
# but wrong for the output document: a legitimate long artifact routinely
# mentions "Unauthorized"/"Not authenticated" mid-sentence. A genuine CLI auth
# failure prints a short banner that STANDS ALONE at the start of a line before
# any work is done. So we scan only the first ~20 non-empty lines and require
# an auth-context banner to be line-leading (leading whitespace/bullet/quote
# punctuation is tolerated — e.g. the CLI's "Not logged in · Please run /login").
# Markdown-QUOTED lines are skipped, not matched: a document about auth
# handling legitimately quotes the banner as an example inside a ``` fence, a
# 4-space/tab indented block, or a > blockquote — a real banner prints at
# column 0 outside any code context. (Bold **...** lead-ins still match: a
# colorized real banner can arrive wrapped; quoting-in-bold is rare.)
# Deliberately excludes a bare "Unauthorized" and a bare "Not authenticated":
# those are prose tokens far more often than banners, and the broad matcher
# still guards the stderr path. This replaces the old whole-file <50-word
# heuristic while preserving deb4669's intent — a long document that merely
# echoes auth phrases mid-body must still pass.
output_contains_auth_banner() {
local file_path="$1"
local line eligible=""
local nonempty=0 in_fence=false
local fence_re='^[[:space:]]*```'
local blockquote_re='^[[:space:]]*>'

[[ -s "$file_path" ]] || return 1

# Collect the window's match-eligible lines, then run the banner regex once.
# `|| [[ -n "$line" ]]` keeps a final line that lacks a trailing newline.
while IFS= read -r line || [[ -n "$line" ]]; do
[[ "$line" =~ ^[[:space:]]*$ ]] && continue # blank: not counted
nonempty=$((nonempty + 1))
(( nonempty > 20 )) && break # window exhausted
if [[ "$line" =~ $fence_re ]]; then # fence marker:
if [[ "$in_fence" == true ]]; then in_fence=false; else in_fence=true; fi
continue # counted, never matched
fi
[[ "$in_fence" == true ]] && continue # inside ``` fence
[[ "$line" == ' '* || "$line" == $'\t'* ]] && continue # indented code
[[ "$line" =~ $blockquote_re ]] && continue # blockquote
eligible+="$line"$'\n'
done < "$file_path"

[[ -n "$eligible" ]] || return 1

grep -qiE \
'^[[:space:]>*•[:punct:]]*(Not logged in|You are not logged in|Please run [^[:space:]]*login|Please sign in|Please log ?in|Login required|Authentication failed|Failed to authenticate|authentication_error|Your organization does not have access|Invalid API key|Session (has )?expired)' \

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve previously handled stdout auth banners

When an auth failure is written to stdout in a form that file_contains_auth_failure() still recognizes, such as Not authenticated. Please run claude login or Please run claude login, this new output-path regex does not match: Not authenticated is no longer an alternative, and Please run [^[:space:]]*login only allows a single no-space token like /login. In those cases validate_agent_artifact now returns 0 for a short auth banner and the bounce path can copy it into the document, which is the failure this gate is meant to prevent.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid treating Markdown headings as auth banners

Because the prefix class includes all [:punct:], valid Markdown headings or list items in the first 20 non-empty lines are reduced to the auth phrase and rejected, e.g. an agent-produced document starting with # Authentication failed handling plan makes validate_agent_artifact return 2 even though it is a normal artifact about auth behavior. This blocks exactly the auth-handling documentation/plans the new tests try to allow; limit the accepted decoration to real banner wrappers instead of every punctuation character.

Useful? React with 👍 / 👎.

<<< "$eligible"
}

file_contains_error_payload() {
local file_path="$1"

Expand All @@ -599,17 +650,17 @@ validate_agent_artifact() {
local output_file="$1"
local stderr_file="$2"
local agent_name="${3:-agent}"
local words

# Fatal: auth-failure text IN THE OUTPUT. Real CLI auth errors are short;
# the <50-word ceiling keeps a long legitimate document that merely
# mentions "Unauthorized" from tripping the gate.
if file_contains_auth_failure "$output_file"; then
words=$(wc -w < "$output_file" | tr -d '\r\n ')
if (( words < 50 )); then
log " ERROR: ${agent_name} returned an authentication failure, not a document. Run \`${agent_name}\` interactively to log in, then re-run."
return 2
fi

# Fatal: an auth-failure BANNER in the output. A real CLI auth error prints
# a short banner that stands alone at the top of its output before doing any
# work, so we head-scan with a strict, line-anchored matcher (A-2). The old
# whole-file <50-word ceiling let an auth-error PAGE longer than 50 words fall
# through to the accept below; anchoring to the head instead catches long
# error pages while still letting a long legitimate document that merely
# echoes auth phrases mid-body pass.
if output_contains_auth_banner "$output_file"; then
log " ERROR: ${agent_name} returned an authentication failure, not a document. Run \`${agent_name}\` interactively to log in, then re-run."
return 2
fi

[[ -s "$output_file" ]] && return 0
Expand Down
Loading
Loading