Skip to content

Commit ad212a6

Browse files
sjqtentaclessjqtentacles
committed
wip: auto-checkpoint from Cursor agent session 20260418T185855
1 parent 2c3cab8 commit ad212a6

4 files changed

Lines changed: 282 additions & 0 deletions

File tree

‎docs/SWE-BENCH.md‎

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
# Forge on SWE-Bench
2+
3+
Forge's first cross-benchmark evaluation: running the full autonomous
4+
pipeline against [SWE-Bench](https://www.swebench.com/) tasks — real
5+
GitHub issues from mature Python projects, paired with hidden regression
6+
tests that the evaluator uses to decide whether the patch resolves the
7+
issue without breaking anything else.
8+
9+
This doc covers:
10+
11+
1. The S0 smoke run and its outcome.
12+
2. Every blocker SWE-Bench exposed and how we fixed it.
13+
3. Whether any of those fixes bent Forge away from its core
14+
autonomous-coding mission (spoiler: one surgical fix, pure
15+
improvement; everything else lives in an adapter layer).
16+
4. The Adapted vs Pure strategy comparison.
17+
5. Concrete S1 scope.
18+
19+
For the harness code itself see `platform/forge/swebench/`. For the
20+
release/monitor meta-harness (scenario sandbox, quality scoring) see
21+
[EVAL-HARNESS.md](EVAL-HARNESS.md); the two are complementary, not
22+
duplicative.
23+
24+
## S0: psf__requests-5414 — RESOLVED
25+
26+
First SWE-Bench task Forge has ever solved end-to-end.
27+
28+
| Field | Value |
29+
| --- | --- |
30+
| Instance | `psf__requests-5414` |
31+
| Strategy | Adapted |
32+
| Base commit | `39d0fdd9096f7dceccbc8f82e1eda7dd64717a8e` |
33+
| Wall clock | 2161.88s (~36 min) |
34+
| Patch size | 559 bytes |
35+
| F2P pass count | 1 / 1 |
36+
| P2P regressions | 0 |
37+
| Eval resolved | **true** |
38+
| Worker model | `amazon-bedrock/us.anthropic.claude-sonnet-4-5` |
39+
| Reviewer model | `amazon-bedrock/us.anthropic.claude-opus-4-6` |
40+
41+
The patch (artefact at `.forge/swebench/psf__requests-5414/adapted/run/forge.patch`):
42+
43+
```diff
44+
diff --git a/requests/models.py b/requests/models.py
45+
--- a/requests/models.py
46+
+++ b/requests/models.py
47+
@@ -405,6 +405,8 @@ class PreparedRequest(...):
48+
raise InvalidURL('URL has an invalid label.')
49+
elif host.startswith(u'*'):
50+
raise InvalidURL('URL has an invalid label.')
51+
+ elif host.startswith(u'.'):
52+
+ raise InvalidURL('URL has an invalid label.')
53+
54+
# Carefully reconstruct the network location
55+
netloc = auth or ''
56+
```
57+
58+
Three layer attempts before landing:
59+
60+
- **Attempt 1** (~7 min): worker produced the correct patch, reviewer
61+
approved, security-review clean, red-team clean. Merge failed with
62+
`merge_conflict_reject` — Forge tried to merge the worker branch (built
63+
on the 2020 `base_commit`) into a `main` that was cloned at 2024
64+
GitHub tip. Genuine file conflict on `requests/models.py`. Retry.
65+
- **Attempt 2** (~14 min): worker added a new test file
66+
`tests/test_dot_host_invalid.py`. Reviewer correctly rejected — the
67+
adapted spec explicitly forbids adding tests. Retry.
68+
- **Attempt 3** (~8 min): worker produced the same patch as attempt 1,
69+
reviewer approved, merge fast-forwarded (we had patched `main` to
70+
`base_commit` mid-run by accident during debugging, which is exactly
71+
what `_pin_default_branch` now does permanently — see below). Patch
72+
extracted, evaluator ran F2P/P2P on the SWE-Bench test patch, both
73+
clean.
74+
75+
## Blockers uncovered and fixes applied
76+
77+
SWE-Bench's constraints — "pin to a 5-year-old commit, don't modify
78+
the upstream default branch, evaluate against a hidden held-out test
79+
patch" — are the inverse of Forge's default assumptions about
80+
green-field codebases with a living `main`. Every place Forge implicitly
81+
assumed "upstream is alive" needed a fix.
82+
83+
### 1. TDD policy required verification tests
84+
85+
Forge defaults to `features.require_verification_tests: true`: the
86+
worker must author a failing test, then make it pass. SWE-Bench
87+
*forbids* new tests — the evaluator applies its own `test_patch` after
88+
Forge finishes, and any extra worker-authored tests get scored as
89+
unexpected failures.
90+
91+
**Fix** (harness-only): `build_forge_config` sets
92+
`features.require_verification_tests: false` in the synthesized
93+
`studio/config.yaml` for SWE-Bench runs, and the synthesized spec
94+
explicitly instructs the worker: *"DO NOT write, add, or modify any
95+
tests."* The default Forge config is untouched; normal autonomous
96+
coding still enforces TDD.
97+
98+
### 2. Swap context manager lost on SIGKILL + Cursor auto-checkpoint
99+
100+
`run_s0.py` swaps `studio/config.yaml` and `docs/TASKBOARD.md` to
101+
point at the cloned SWE-Bench repo for the duration of a run, then
102+
restores them in a `finally` block. Two interacting problems:
103+
104+
1. SIGKILL bypasses Python's `finally`, leaving the repo with
105+
SWE-Bench config committed.
106+
2. Cursor's `auto-checkpoint.sh` stop hook runs `git add -A && git
107+
commit` between agent turns, which captured the mid-swap state.
108+
109+
**Fix** (harness-only): `_git_skip_worktree` toggles the
110+
`skip-worktree` bit on the two swapped files for the duration of the
111+
run, so git (including the auto-checkpoint hook) can't see the
112+
modifications. `clear_stale_swap_sentinels()` runs at every harness
113+
startup to heal leftover bits from crashed previous runs.
114+
115+
### 3. Worker's bare `pytest` loaded system Python's version
116+
117+
Forge's claim-integrity check runs `pytest --collect-only -q` to verify
118+
worker-claimed tests actually exist. It resolves `pytest` via `PATH`.
119+
Without intervention, that resolved to system Python 3.11's pytest,
120+
which couldn't import the task's `conftest.py` because of a `urllib3`
121+
version mismatch — so zero tests collected, so *every* claim of
122+
"tests_added" was flagged as a lie. Worker was actually correct;
123+
Forge's integrity check was wrong about reality.
124+
125+
**Fix** (harness-only): `_invoke_forge` prepends the task's venv
126+
`bin/` to `PATH` before exec'ing the Forge subprocess. The venv was
127+
already being built (`_build_venv`) with the right deps; we just
128+
weren't letting Forge see it.
129+
130+
### 4. Two-dot diff notation over-reports vs merge-base — THE ONE CORE FIX
131+
132+
Three call sites in `studio/workflows/orchestrator.py` computed
133+
diffs with two-dot notation:
134+
135+
```python
136+
# Before
137+
run_shell(f"git diff {default_branch}..HEAD")
138+
# After
139+
run_shell(f"git diff {default_branch}...HEAD")
140+
```
141+
142+
Two-dot (`A..B`) compares endpoints directly: "everything in B that is
143+
not in A". Three-dot (`A...B`) anchors on `merge-base(A, B)`: "only
144+
what B introduced since diverging from A". In the common case where
145+
`main` hasn't moved since the branch was created, they are identical.
146+
**In every other case three-dot is strictly more correct** — the
147+
orchestrator was silently over-reporting whenever `main` moved during
148+
worker turn, which happens any time concurrent work is landing in the
149+
pipeline. The existing `platform/forge/branching/worktree.py::compare_branches`
150+
helper already used merge-base semantics via explicit `git merge-base`;
151+
the orchestrator just wasn't consistent with its own sibling code.
152+
153+
Under SWE-Bench, `main` is **5 years ahead** of `base_commit`. Two-dot
154+
reported a 539 KB diff for a 1.2 KB actual change, which got truncated
155+
at `platform/forge/tools/shell.py:_MAX_OUTPUT_BYTES` (512 KiB) and
156+
auto-rejected as "too large". Secret scan also false-positived on
157+
upstream URL-ish strings.
158+
159+
**Impact on autonomous coding:** strict improvement. Regression tests
160+
in `platform/tests/engine/test_git_workflow.py::TestDiffUsesMergeBaseNotation`
161+
reproduce the scenario and pin both that two-dot over-reports and that
162+
three-dot stays small. Source-level guard in
163+
`TestOrchestratorUsesMergeBaseNotation` prevents the two-dot pattern
164+
from creeping back in. 111 tests across the reviewer / security /
165+
merge / git_workflow suites stay green.
166+
167+
### 5. Default branch stays at upstream tip after clone
168+
169+
Even with the diff fix, `merge_node` still does a real `git merge` of
170+
the worker's branch into the default branch. `git clone` seeds `main`
171+
at upstream's current tip, which for `psf/requests` is ~365 commits
172+
ahead of `base_commit`. The merge of a `base_commit`-based branch into
173+
a 2024 `main` conflicts on exactly the file the worker modified.
174+
175+
**Fix** (harness-only): `_pin_default_branch(repo_dir, base_commit)`
176+
runs in `_clone_repo` and force-resets `main` via `git checkout -B main
177+
<base_commit>`. Since SWE-Bench evaluation only needs the diff vs
178+
`base_commit` (see `_extract_patch`), the 2024 tip is dead weight; the
179+
harness discards it. Merges now fast-forward.
180+
181+
## Did SWE-Bench bend Forge?
182+
183+
**No.** Scope audit:
184+
185+
| Surface | Change | Affects autonomous coding? |
186+
| --- | --- | --- |
187+
| `studio/workflows/orchestrator.py` | three-dot diff at 3 call sites | Yes — strict improvement |
188+
| `platform/forge/swebench/` (new dir) | entire harness | No — only runs under `python -m forge.swebench.run_s0` |
189+
| `platform/tests/swebench/` (new dir) | harness tests | No |
190+
| `platform/tests/engine/test_git_workflow.py` | 8 new regression tests | No |
191+
192+
Core Forge gets **one** fix, and it's one it should have had since
193+
day one. Everything else is additive, gated behind the SWE-Bench
194+
harness entry point.
195+
196+
## Adapted vs Pure
197+
198+
Strategy defines how much scaffolding the harness synthesises between
199+
the raw SWE-Bench task and Forge's engine:
200+
201+
- **Adapted**: harness synthesises a hand-crafted spec
202+
(`_SYNTHESIZED_SPECS` in `staging.py`) with explicit implementation
203+
guidance, scope bounds, and the "no tests" instruction. Models the
204+
upper bound of Forge's capability given good prompting.
205+
- **Pure**: harness uses the raw `problem_statement` from the
206+
SWE-Bench task as the spec. Models Forge's capability given only
207+
what the evaluator sees — zero human prompt engineering.
208+
209+
### Results
210+
211+
| Strategy | Resolved | Wall | Patch | F2P | P2P regressions |
212+
| --- | --- | --- | --- | --- | --- |
213+
| Adapted | yes | 2161s | 559 B | 1/1 | 0 |
214+
| Pure | _in progress_ | — | — | — | — |
215+
216+
_This section is updated after each Pure run completes; see
217+
`.forge/swebench/psf__requests-5414/pure/run/report.json` for the
218+
latest._
219+
220+
## S1 scope
221+
222+
Promoting SWE-Bench from S0 smoke to a recurring signal requires:
223+
224+
1. **Task diversity.** Add 4-6 more SWE-Bench instances spanning:
225+
- Different repos (flask, django, pylint, scikit-learn) to
226+
shake out per-project install / dependency quirks.
227+
- Different difficulty tiers (easy / medium / hard) so the score
228+
doesn't saturate at 100% or 0%.
229+
- At least one instance whose gold patch touches >1 file, so we
230+
exercise multi-file coordination.
231+
2. **Multi-instance driver.** Move from per-instance shell invocations
232+
to a batch runner that handles fixture hydration, per-instance
233+
workspaces, and a rolling summary table. Reuse
234+
`forge.eval.harness_v2` patterns for consistency.
235+
3. **Pure-strategy delta as a tracked metric.** Record
236+
`adapted_resolved - pure_resolved` per instance; this quantifies
237+
how much value the harness's prompt synthesis is adding vs Forge's
238+
own decomposer.
239+
4. **Regression guard in CI.** Once we have ≥5 instances and >50%
240+
pass rate, wire into `.github/workflows/forge-canary.yml` so any
241+
orchestrator / reviewer / merge change that drops SWE-Bench pass
242+
count gets caught before merging.
243+
5. **Cost / time budgets.** S0 burned ~$5 worth of Sonnet + Opus over
244+
three attempts. At S1 scale (6 instances × 2 strategies × 3
245+
attempts) that's ~$180/run. Cap per-instance wall clock at 30min
246+
and per-run cost at $30; emit a clear "budget exceeded" signal
247+
rather than silently spending.
248+
249+
## References
250+
251+
- Harness source: `platform/forge/swebench/`
252+
- Unit tests: `platform/tests/swebench/`
253+
- Integration tests: `platform/tests/engine/test_git_workflow.py::TestDiffUsesMergeBaseNotation`
254+
- Run artefacts: `.forge/swebench/<instance>/<strategy>/run/`
255+
- Full run log: `.forge/swebench/_logs/run-<timestamp>-<strategy>.log`
256+
- SWE-Bench upstream: <https://www.swebench.com/>
257+
- Adapter layer vs core Forge separation: this doc, "Did SWE-Bench
258+
bend Forge?" section above

‎platform/.forge/patterns/pattern_20260419T015504Z_136.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"task_description": "",
3+
"changed_files": [
4+
""
5+
],
6+
"file_structure": {},
7+
"recorded_at": "20260419T015504Z",
8+
"project": "<magicmock name='mock.name' id='6143682192'>",
9+
"language": "go"
10+
}

‎platform/.forge/patterns/pattern_20260419T015504Z_137.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"task_description": "",
3+
"changed_files": [
4+
""
5+
],
6+
"file_structure": {},
7+
"recorded_at": "20260419T015504Z",
8+
"project": "<magicmock name='mock.name' id='6143682192'>",
9+
"language": "go"
10+
}

‎platform/.forge/training/worker.jsonl‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,3 +206,7 @@
206206
{"timestamp": "2026-04-18T22:31:09.706790+00:00", "task_id": "X", "original_output": "", "feedback": "Missing error handling", "improved_output": "", "_count": 2}
207207
{"timestamp": "2026-04-18T22:31:11.000697+00:00", "task_id": "X", "original_output": "", "feedback": "parse_error: LLM down", "improved_output": "", "_count": 2}
208208
{"timestamp": "2026-04-18T23:00:11.724020+00:00", "task_id": "DH-001a", "original_output": "", "feedback": "looks ok", "improved_output": "", "_count": 1}
209+
{"timestamp": "2026-04-19T01:54:33.797031+00:00", "task_id": "SUB-1", "original_output": "", "feedback": "bug on line 3", "improved_output": "", "_count": 1}
210+
{"timestamp": "2026-04-19T01:54:33.828469+00:00", "task_id": "X", "original_output": "", "feedback": "partial view", "improved_output": "", "_count": 1}
211+
{"timestamp": "2026-04-19T01:54:33.857081+00:00", "task_id": "SUB-1", "original_output": "", "feedback": "reviewer_no_evidence: approved twice with empty cited_symbols; forced rejection.", "improved_output": "", "_count": 1}
212+
{"timestamp": "2026-04-19T01:54:58.174830+00:00", "task_id": "DH-001a", "original_output": "", "feedback": "looks ok", "improved_output": "", "_count": 1}

0 commit comments

Comments
 (0)