Skip to content

Commit 3148aee

Browse files
AbirAbbasclaude
andauthored
feat: post-PR CI gate — watch, legitimate-fix, promote to ready (#57)
* feat(schemas): add check_ci config + ci_fixer role + CI-gate result schemas Introduces the configuration surface for the post-PR CI gate without changing runtime behaviour. - BuildConfig.check_ci (default true) plus max_ci_fix_cycles, ci_wait_seconds, and ci_poll_seconds caps. ExecutionConfig mirrors these so they round-trip through to_execution_config_dict(). - New ci_fixer model role registered in ROLE_TO_MODEL_FIELD so the CI fixer picks up the runtime base default and can be overridden via models.ci_fixer like any other role. - New Pydantic models — CIFailedCheck, CIWatchResult, CIFixResult — for the watcher and fixer result envelopes. BuildResult gains ci_gate_results so callers can inspect the gate outcome alongside pr_results. Tests cover the new role property, the new config defaults, and round-trip through BuildConfig -> ExecutionConfig. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ci_gate): deterministic gh-CLI watcher + mark_pr_ready helper The post-PR CI gate needs to poll GitHub Actions on a draft PR, fetch the failed-job log tail, and promote the PR to ready-for-review. All of that is shell-and-state, no LLM, so it lives in a standalone helper module that the reasoners thinly wrap. watch_pr_checks accepts injectable runner/sleep/now callables so the polling loop is unit-testable without invoking gh, sleeping in real time, or hitting GitHub. Handles the corner where `gh pr checks` exits non-zero with a valid JSON body (its convention when any check is failing) by trusting the body. Tests cover happy-path pass, failure with log capture, multi-poll until conclusive, wall-clock timeout, no-checks-on-PR, gh-failure-with-no-payload, and the mark_pr_ready success/failure paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(reasoners): add run_ci_watcher and run_ci_fixer run_ci_watcher is a thin reasoner around watch_pr_checks — no LLM, just polling + log capture. run_ci_fixer is the agentic counterpart: it gets the failing checks (with truncated logs) and must produce a legitimate fix committed and pushed to the PR's integration branch. The CI-fixer system prompt explicitly forbids the workarounds the model would otherwise reach for: pytest.skip / xfail / it.skip, commenting tests out, deleting tests, loosening assertions to make red green, swallowing errors with try/except: pass, disabling CI jobs, snapshotting the bug, or mocking the unit under test. The agent is told to fix the production code and asked to enumerate rejected workarounds in the response as an audit trail. Editing the test is allowed only when it asserts something the spec doesn't require, with explicit justification. Both reasoners are exposed via swe_af.fast wrappers and the fast-router reasoner-count test is updated from 8 to 10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(build): wire post-PR CI gate into the build pipeline After the draft PR is opened, _run_ci_gate runs a bounded watch -> fix -> repush loop and promotes the PR with `gh pr ready` only once CI is green. Wired into both the multi-repo and single-repo PR paths so all PRs go through the same gate. Each repo's gate result is captured in BuildResult.ci_gate_results for visibility. The loop is bounded by cfg.max_ci_fix_cycles (default 2) and cfg.ci_wait_seconds (default 1500s per watch). Failure modes are explicit final_status values — failed_exhausted, fixer_gave_up, timed_out, error, no_checks — so callers can distinguish "gave up after N tries" from "never produced a fix" or "CI never ran". When the gate gives up, the PR is left in draft so a human reviewer sees it needs attention. Default behaviour is check_ci=true. Setting check_ci=false reverts to the legacy "create draft PR and exit" flow. README documents the gate behaviour, the no-workarounds contract, and the four config knobs (check_ci, max_ci_fix_cycles, ci_wait_seconds, ci_poll_seconds). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent d9eb2fe commit 3148aee

10 files changed

Lines changed: 1306 additions & 6 deletions

File tree

README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,25 @@ Requirements:
512512
- `GH_TOKEN` in `.env` with `repo` scope
513513
- Repo access for that token
514514

515+
### Post-PR CI gate
516+
517+
After SWE-AF pushes the integration branch and opens a draft PR, it watches
518+
GitHub Actions on that PR until checks are conclusive. If they fail, a
519+
bounded fix-and-repush loop runs an agent that is explicitly forbidden from
520+
silencing tests (no `pytest.skip`, no `xfail`, no commenting tests out, no
521+
loosening assertions) — it must produce a legitimate fix in the production
522+
code and push a new commit. When CI is green, the PR is promoted from draft
523+
to ready-for-review via `gh pr ready`.
524+
525+
Configuration on `BuildConfig`:
526+
527+
| Field | Default | Purpose |
528+
|---|---|---|
529+
| `check_ci` | `true` | Run the post-PR CI gate. Set `false` to return immediately after the draft PR is created. |
530+
| `max_ci_fix_cycles` | `2` | Cap on watch → fix → repush iterations after the initial push. |
531+
| `ci_wait_seconds` | `1500` | Wall-clock cap per `gh pr checks` watch (25 min). |
532+
| `ci_poll_seconds` | `30` | Poll interval for `gh pr checks`. |
533+
515534
## API Reference
516535

517536
<details>

swe_af/app.py

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
from swe_af.reasoners.schemas import PlanResult, ReviewResult
1919

2020
from agentfield import Agent
21+
from swe_af.execution.ci_gate import mark_pr_ready
2122
from swe_af.execution.envelope import unwrap_call_result as _unwrap
2223
from swe_af.execution.schemas import (
2324
BuildConfig,
@@ -169,6 +170,136 @@ def _run() -> str:
169170
)
170171

171172

173+
async def _run_ci_gate(
174+
*,
175+
repo_path: str,
176+
pr_number: int,
177+
pr_url: str,
178+
integration_branch: str,
179+
base_branch: str,
180+
cfg: BuildConfig,
181+
resolved_models: dict,
182+
goal: str,
183+
completed_issues: list[dict],
184+
) -> dict:
185+
"""Watch CI on the freshly-pushed draft PR; fix-and-repush if it fails;
186+
promote to ready-for-review when green.
187+
188+
Returns a summary dict the build can attach to its response. Bounded by
189+
``cfg.max_ci_fix_cycles`` and ``cfg.ci_wait_seconds`` per watch.
190+
"""
191+
attempts: list[dict] = []
192+
last_watch: dict | None = None
193+
194+
for cycle in range(cfg.max_ci_fix_cycles + 1):
195+
app.note(
196+
f"CI gate: watch cycle {cycle + 1} for PR #{pr_number}",
197+
tags=["ci_gate", "watch"],
198+
)
199+
watch = _unwrap(await app.call(
200+
f"{NODE_ID}.run_ci_watcher",
201+
repo_path=repo_path,
202+
pr_number=pr_number,
203+
wait_seconds=cfg.ci_wait_seconds,
204+
poll_seconds=cfg.ci_poll_seconds,
205+
), "run_ci_watcher")
206+
last_watch = watch
207+
status = watch.get("status", "error")
208+
209+
if status in ("passed", "no_checks"):
210+
ok, msg = mark_pr_ready(repo_path=repo_path, pr_number=pr_number)
211+
app.note(
212+
f"CI gate: {status}{msg}",
213+
tags=["ci_gate", "ready" if ok else "ready_failed"],
214+
)
215+
return {
216+
"final_status": "passed" if status == "passed" else "no_checks",
217+
"promoted_to_ready": ok,
218+
"promote_message": msg,
219+
"fix_attempts": attempts,
220+
"watch": watch,
221+
}
222+
223+
if status in ("timed_out", "error"):
224+
app.note(
225+
f"CI gate: {status} — leaving PR in draft. {watch.get('summary', '')}",
226+
tags=["ci_gate", status],
227+
)
228+
return {
229+
"final_status": status,
230+
"promoted_to_ready": False,
231+
"promote_message": "",
232+
"fix_attempts": attempts,
233+
"watch": watch,
234+
}
235+
236+
# status == "failed"
237+
if cycle >= cfg.max_ci_fix_cycles:
238+
app.note(
239+
f"CI gate: exhausted {cfg.max_ci_fix_cycles} fix cycle(s) — "
240+
"leaving PR in draft",
241+
tags=["ci_gate", "exhausted"],
242+
)
243+
return {
244+
"final_status": "failed_exhausted",
245+
"promoted_to_ready": False,
246+
"promote_message": "",
247+
"fix_attempts": attempts,
248+
"watch": watch,
249+
}
250+
251+
failed_checks = watch.get("failed_checks", [])
252+
app.note(
253+
f"CI gate: fix attempt {cycle + 1}/{cfg.max_ci_fix_cycles} — "
254+
f"{len(failed_checks)} failing check(s)",
255+
tags=["ci_gate", "fix"],
256+
)
257+
fix = _unwrap(await app.call(
258+
f"{NODE_ID}.run_ci_fixer",
259+
repo_path=repo_path,
260+
pr_number=pr_number,
261+
pr_url=pr_url,
262+
integration_branch=integration_branch,
263+
base_branch=base_branch,
264+
failed_checks=failed_checks,
265+
iteration=cycle + 1,
266+
max_iterations=cfg.max_ci_fix_cycles,
267+
goal=goal,
268+
completed_issues=completed_issues,
269+
previous_attempts=attempts,
270+
model=resolved_models.get("ci_fixer_model", resolved_models.get("coder_model", "")),
271+
permission_mode=cfg.permission_mode,
272+
ai_provider=cfg.ai_provider,
273+
), "run_ci_fixer")
274+
attempts.append(fix)
275+
276+
if not fix.get("pushed"):
277+
app.note(
278+
f"CI gate: fixer did not push ({fix.get('summary', 'no summary')}) — "
279+
"leaving PR in draft",
280+
tags=["ci_gate", "fixer_no_push"],
281+
)
282+
return {
283+
"final_status": "fixer_gave_up",
284+
"promoted_to_ready": False,
285+
"promote_message": "",
286+
"fix_attempts": attempts,
287+
"watch": watch,
288+
}
289+
290+
# Pushed — loop back and watch again. GitHub may take a moment to
291+
# register the new run; watcher's poll_seconds covers that.
292+
293+
# Loop fell through (shouldn't happen because the failed branch returns).
294+
return {
295+
"final_status": "loop_exhausted",
296+
"promoted_to_ready": False,
297+
"promote_message": "",
298+
"fix_attempts": attempts,
299+
"watch": last_watch or {},
300+
}
301+
302+
172303
@app.reasoner()
173304
async def build(
174305
goal: str,
@@ -622,6 +753,7 @@ async def build(
622753

623754
# 4. PUSH & DRAFT PR (if repo has a remote and PR creation is enabled)
624755
pr_results: list[RepoPRResult] = []
756+
ci_gate_results: list[dict] = []
625757
build_summary = (
626758
f"{'Success' if success else 'Partial'}: {completed}/{total} issues completed"
627759
+ (f", verification: {verification.get('summary', '')}" if verification else "")
@@ -676,6 +808,25 @@ async def build(
676808
f"Draft PR created for {ws_repo.repo_name}: {pr_r.get('pr_url')}",
677809
tags=["build", "github_pr", "complete"],
678810
)
811+
if cfg.check_ci and pr_r.get("pr_number"):
812+
gate = await _run_ci_gate(
813+
repo_path=ws_repo.absolute_path,
814+
pr_number=pr_r.get("pr_number", 0),
815+
pr_url=pr_r.get("pr_url", ""),
816+
integration_branch=repo_integration_branch,
817+
base_branch=repo_base_branch,
818+
cfg=cfg,
819+
resolved_models=resolved,
820+
goal=goal,
821+
completed_issues=[
822+
r for r in dag_result.get("completed_issues", [])
823+
if not r.get("repo_name") or r.get("repo_name") == ws_repo.repo_name
824+
],
825+
)
826+
ci_gate_results.append({
827+
"repo_name": ws_repo.repo_name,
828+
**gate,
829+
})
679830
except Exception as e:
680831
pr_results.append(RepoPRResult(
681832
repo_name=ws_repo.repo_name,
@@ -770,6 +921,25 @@ async def build(
770921
pr_url=pr_url,
771922
pr_number=pr_result.get("pr_number", 0),
772923
))
924+
if cfg.check_ci and pr_result.get("pr_number"):
925+
gate = await _run_ci_gate(
926+
repo_path=repo_path,
927+
pr_number=pr_result.get("pr_number", 0),
928+
pr_url=pr_url,
929+
integration_branch=git_config["integration_branch"],
930+
base_branch=base_branch,
931+
cfg=cfg,
932+
resolved_models=resolved,
933+
goal=goal,
934+
completed_issues=dag_result.get("completed_issues", []),
935+
)
936+
ci_gate_results.append({
937+
"repo_name": (
938+
_repo_name_from_url(cfg.repo_url)
939+
if cfg.repo_url else "repo"
940+
),
941+
**gate,
942+
})
773943
except Exception as e:
774944
app.note(f"PR creation failed: {e}", tags=["build", "github_pr", "error"])
775945

@@ -793,6 +963,7 @@ async def build(
793963
summary=f"{'Success' if success else 'Partial'}: {completed}/{total} issues completed"
794964
+ (f", verification: {verification.get('summary', '')}" if verification else ""),
795965
pr_results=pr_results,
966+
ci_gate_results=ci_gate_results,
796967
).model_dump()
797968

798969

0 commit comments

Comments
 (0)