[WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估 (#1947)

* feat(evals): add SWE-bench Lite eval (lm-eval generation + swebench harness scoring, Modal-capable)

Add a SWE-bench Lite accuracy eval that generates patches via the lm-eval
harness and scores them with the official swebench evaluation harness.

- utils/evals/swebench_lite.yaml: lm-eval task config for SWE-bench Lite
  generation (prompt/doc-to-text, generation kwargs, dataset wiring).
- utils/evals/swebench_score.py: post-processing + scoring. Extracts model
  patches from lm-eval output, feeds them to the swebench harness, and emits
  a "resolved" rate. Supports running the harness locally or on Modal via
  SWEBENCH_USE_MODAL (Modal pass-through so scoring can run off-box).
- utils/collect_eval_results.py: extract_lm_metrics learns a "resolved"
  filter branch so the swebench resolved metric is collected alongside the
  existing lm-eval metrics.
- utils/evals/thresholds.json: add the swebench_lite threshold entry.
- utils/evals/EVALS.md: document the SWE-bench Lite eval and how scoring works.
- benchmarks/benchmark_lib.sh: add run_swebench_eval, _install_swebench_deps,
  maybe_run_eval, and Modal pass-through. run_eval now picks a per-scenario
  default framework (agentic-coding -> swebench, fixed-seq-len -> lm-eval);
  an explicit EVAL_FRAMEWORK env var or --framework arg overrides the default.
  EVAL_TASKS_DIR selects the task yaml.
- utils/evals/test_swebench_eval.py, utils/evals/test_run_eval_dispatch.py:
  tests for the scorer and the scenario/framework dispatch precedence.

* feat(evals): agentic-scenario eval selection + routing (swebench on agentic configs)

Wire the SWE-bench Lite eval into the sweep matrix so it runs on agentic
coding configs, and route it through e2e-tests.

- utils/matrix_logic/generate_sweep_configs.py: add mark_eval_entries and
  mark_all_eval_entries. For agentic configs these mark exactly one eval
  entry per (model, runner, framework, precision) group at the highest
  concurrency, single-node only, so each unique agentic config gets one
  swebench eval run rather than one per concurrency point.
- utils/matrix_logic/test_generate_sweep_configs.py: add
  test_marks_agentic_entry_for_swebench and update TestMarkAllEvalEntries
  to cover the agentic marking behavior.
- .github/workflows/e2e-tests.yml: add the agentic-eval-config bucket, a
  test-sweep-agentic-evals job, and make collect-evals depend on it. The
  AGENTIC_EVAL filter (agentic + no prefill + run-eval) selects the eval
  entries; the throughput AGENTIC filter (agentic + not run-eval) excludes
  them so throughput and eval runs don't collide.
- benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh: add the eval hook so
  the recipe triggers the agentic swebench eval.

* fix(evals): swebench Modal uses --max_workers (no --parallelism in 4.1.0) + bootstrap Modal creds from env

swebench 4.1.0 exposes --max_workers in both Docker and Modal modes; --parallelism
does not exist. Fix run_harness() to emit --max_workers in the Modal branch.

Add _ensure_modal_credentials() to benchmark_lib.sh: swebench's credential
check only looks for ~/.modal.toml, but CI supplies MODAL_TOKEN_ID/
MODAL_TOKEN_SECRET env vars (GitHub secret). The helper bootstraps the file
from the env vars when the file is absent, so the harness check passes. Called
in run_swebench_eval() right after _install_swebench_deps, scoring path only.

Update the Modal test name and assertions, the run_swebench_eval docstring,
and the EVALS.md knobs bullet to document the credential bootstrapping.

* feat(evals): eval-only gating for all single-node agentic recipes

Apply the EVAL_ONLY=true if/else gating pattern (already present in
kimik2.5_fp4_b300.sh) to the remaining 24 single-node agentic recipes in
benchmarks/single_node/agentic/. In eval-only mode each recipe skips the
multi-turn agentic replay and calls maybe_run_eval "$PORT" against the live
server; run_eval auto-selects swebench for the agentic-coding scenario.
The deprecated/ subdirectory was not touched.

* feat(evals): thread Modal credentials + SWEBENCH_USE_MODAL into eval job env

GitHub secrets MODAL_TOKEN_ID/MODAL_TOKEN_SECRET are now available; bootstrap
into ~/.modal.toml happens in benchmark_lib.sh:_ensure_modal_credentials.
SWEBENCH_USE_MODAL is only read by swebench-path functions, so it is inert for
lm-eval/gsm8k jobs.

* fix(evals): adapt agentic eval wiring to AgentX v1.0

- Re-sync test-sweep-agentic-evals inputs with main's test-sweep-agentic:
  offloading -> kv-offloading + kv-offload-backend + total-cpu-dram-gb.
- Add EVAL_ONLY/maybe_run_eval tail gating to the agentic recipes AgentX
  v1.0 added (dsv4_fp4_b200_sglang, dsv4_fp4_b300_sglang, minimaxm3_fp8_h100/
  h200/mi300x/mi325x) so eval-only runs skip the replay like the others.
- test_run_eval_dispatch: set KV_OFFLOADING=none so the new source-time
  agentic guard in benchmark_lib.sh is satisfied (dispatch logic unaffected).

* feat(evals): eval-limit smoke knob + robust Modal credential HOME handling

Add EVAL_LIMIT env var to run_lm_eval() so --limit N is appended to the
lm_eval invocation when set, enabling small smoke runs (e.g. 10 instances)
without touching the full ~300-instance swebench suite. Wire the knob through
benchmark-tmpl.yml (new eval-limit input + EVAL_LIMIT env) and e2e-tests.yml
(both workflow_dispatch and workflow_call inputs; passed through to
test-sweep-evals and test-sweep-agentic-evals with: blocks). Document the
variable in utils/evals/EVALS.md.

Harden _ensure_modal_credentials against b300 slurm/pyxis containers where
--export=ALL propagates the HOST's HOME into the container; if HOME is unset,
mkdir -p fails, or the directory isn't writable, remap HOME to
/tmp/inferencex-modal-home before writing ~/.modal.toml. Remap is scoped to
the write path (SWEBENCH_USE_MODAL=true, file absent, tokens present).

Tests: functional shim tests for --limit presence/absence; HOME-remap tests
covering writable home (no remap), read-only parent (remap + 600 perms), and
non-writable existing dir (remap); and a no-op test when SWEBENCH_USE_MODAL=false.

* fix(evals): gate agentic eval marking to evals-only/all-evals + fix empty SWEBENCH_NAMESPACE arg

- Add `include_agentic: bool = False` to `mark_eval_entries`; wrap the
  `ag_sn_groups` agentic-marking block in `if include_agentic:` so that
  default sweeps no longer set `run-eval: true` on any agentic entry.
  The e2e-tests.yml AGENTIC filter (`not x.get('run-eval', False)`) then
  routes all agentic entries to the throughput job, restoring main parity.
- Pass `include_agentic=args.evals_only or args.all_evals` in `main()` so
  --evals-only and --all-evals continue to mark and select agentic entries.
- Replace `${SWEBENCH_NAMESPACE+--namespace "$SWEBENCH_NAMESPACE"}` with an
  `ns_args` array in `run_swebench_eval`; when `SWEBENCH_NAMESPACE=""` the
  old form word-split to a bare `--namespace` (argparse error); the array
  form safely expands `--namespace ""` or nothing when unset.
- Tests: `test_marks_agentic_entry_for_swebench` updated to pass
  `include_agentic=True`; new `test_default_mode_does_not_mark_agentic`
  asserts zero agentic entries marked in default mode; new ns_args unit
  tests cover unset/empty/value cases plus a static assertion that the old
  pattern is gone from benchmark_lib.sh.

* fix(evals): register swebench task via --include_path (pinned lm-eval KeyErrors on unregistered task-name paths)

The pinned lm-eval (0.4.9.2, ref b315ef3) crashes with
KeyError: '<task_name>' in pretty_print_task (tasks/__init__.py:681) when
--tasks is given a file path to an external YAML whose task: name is not in
lm-eval's bundled registry.  gsm8k/gpqa_diamond are immune because those
names exist in the bundled registry; swebench_lite is not.

Fix: in run_lm_eval(), add optional EVAL_INCLUDE_PATH support — when set,
injects --include_path "$EVAL_INCLUDE_PATH" just before --tasks; inert when
unset (gsm8k/gpqa production invocations are byte-identical).

In run_swebench_eval(), switch the generation call from
  EVAL_TASKS_DIR="$yaml_path"     (path form → KeyError)
to
  EVAL_TASKS_DIR="$task_name"     (name form)
  EVAL_INCLUDE_PATH="$(dirname "$yaml_path")"   (registers the dir)
with save/restore of both vars so EVAL_INCLUDE_PATH does not leak to
subsequent lm-eval invocations.  The dataset_path-from-YAML derivation
(awk over yaml_path) is unchanged — generation and scoring remain in lockstep.

Tests: two shim-based dynamic tests (EVAL_INCLUDE_PATH set/unset → flag
present/absent in argv; --tasks carries name vs. yaml path) and one static
assertion that run_swebench_eval source contains EVAL_INCLUDE_PATH wiring.

* fix(evals): sanitize Modal token env vars (CI secrets pasted with trailing newline fail validation)

Live probe proved it: MODAL_TOKEN_SECRET secret has a trailing whitespace char;
raw auth fails ('Token validation failed'), whitespace-stripped auth succeeds.
Strip whitespace/quotes and re-export in _ensure_modal_credentials so both the
modal client (env) and the bootstrapped ~/.modal.toml are clean.

* fix(evals): scoring timeout guard + artifact preservation on eval failure

- run_swebench_eval: wrap scoring in timeout ${SWEBENCH_SCORE_TIMEOUT:-7200}s.
  The overnight 300-instance run stalled ~7h in Modal image builds and held the
  b300 allocation until the slurm wall; a stalled backend now fails fast.
- maybe_run_eval: always stage eval artifacts (append_lm_eval_summary) even when
  the eval fails, then propagate the rc — samples/predictions survive for
  diagnosis instead of dying in the job sandbox.

* feat(evals): --predictions-file mode in swebench_score (agentic generation input)

Agent harnesses (SWE-agent / mini-swe-agent) emit standard predictions.jsonl
directly; this bypasses lm-eval samples parsing and feeds the existing Modal
scoring + results pipeline unchanged. Groundwork for agentic swebench.

* feat(evals): agentic SWE-bench generation via mini-swe-agent + Modal sandboxes

SWEBENCH_GEN_MODE=agentic runs a real agent loop per instance instead of the
single-shot prompt: mini-swe-agent (2.4.5) drives the local OpenAI-compatible
endpoint; each instance's shell executes in a Modal sandbox (swe-rex[modal],
official swebench per-instance images -- no docker needed on the GPU node).
preds.json feeds the existing Modal scoring via --predictions-file (which now
also accepts the dict-keyed preds.json format directly).

- benchmark_lib.sh: _run_swebench_agentic_generation (config overlay, slice via
  EVAL_LIMIT, workers/step/timeout knobs), _install_swebench_agent_deps
  (mini-swe-agent==2.4.5 + swe-rex[modal]==1.4.0), gen-mode branch in
  run_swebench_eval feeding scoring via score_input array.
- swebench_score.py: --predictions-file accepts dict preds.json or JSONL.
- workflows: swebench-gen-mode input threaded e2e-tests -> benchmark-tmpl env.
- tests: shim-driven agentic-generation test + predictions-file format tests.

Single-shot remains the default; agentic is the real SWE-bench setting.

* fix(evals): mini-swe-agent import banner pollutes config-path capture

Fresh installs print a multi-line version banner on import; take only the last
stdout line and validate it is a file. Shim test now emulates the banner.

* fix(evals): raise swe-rex Modal sandbox startup timeout for agentic mode

mini's default startup_timeout=60s is consumed by the cold GB-scale swebench
image pull alone ('Runtime did not start within 0s'). Default 900s via
SWEBENCH_AGENT_STARTUP_TIMEOUT; command timeout 300s (mini default 60s is too
tight for running repo test suites) via SWEBENCH_AGENT_CMD_TIMEOUT.

* feat(evals): preserve agent trajectories + predictions as run artifacts

Trajectories are the primary forensic artifact for agent tuning; they
previously died with the job's temp dir. Copy *.traj* flat into the eval
output (append_lm_eval_summary flattens *.json* into the workspace root),
upload via new globs, and clean up post-upload.

* feat(evals): trajectory-forensics guidance in the agent template

Findings from 10-trajectory deep-dive (first-10 Lite, DSv4):
- 3/5 unresolved agents submitted without ever running the failing test
- 1 agent had the CORRECT fix on disk at step 31, burned 44 steps fighting an
  unfixable sandbox C-extension build, and hit the step cap without submitting
- CoT leaks into visible content (deepseek_v4 reasoning parser init failure,
  recipe-side follow-up) -- 'execute over prose' guidance mitigates

Replace the static config heredoc with a runtime merger that appends targeted
guidance to mini's instance_template: verify-before-submit, build-failure
escape hatch, submission discipline, step-budget framing. Single merged config
replaces the dual -c chain.

* fix(swebench): stop leaking Modal sandboxes to the 1h runtime_timeout

Every agent sandbox was billing a full hour for ~7-minute instances
(observed: batches dying at 59m59s on the Modal dashboard). Three leaks:

- mini-swe-agent 2.4.5 process_instance() never calls env.stop(), even on
  success, so every sandbox lives until runtime_timeout (3600s default).
- swe-rex 1.4.0 ModalDeployment.stop() has its poll check inverted: it
  terminates only sandboxes that already exited and skips running ones.
- ModalDeployment.start() leaks the sandbox when the runtime never comes
  alive (the startup-timeout failure mode).

Fix: _patch_swebench_agent_cleanup() patches the installed files at dep
install (idempotent, anchor-checked against the pinned versions) so
sandboxes terminate the moment their instance finishes; a post-generation
workspace sweep reaps anything that slips through (crashed workers, outer
timeout kills; SWEBENCH_SANDBOX_SWEEP=0 disables for tests); and the
merged config now sets runtime_timeout explicitly
(SWEBENCH_AGENT_RUNTIME_TIMEOUT, default 3600) as a pure backstop.

No agent-visible behavior change: cleanup happens after instance
completion, so resolved-rate comparisons across runs stay clean.

* fix(swebench): correct resolved-rate denominator; submit tree diff on budget exhaustion

Run-1/3 findings (50 instances, tuned template):

- Metric bug: the harness report's total_instances is the full dataset size
  (300) even with EVAL_LIMIT=50, so a 32/50 (64%) run was published as
  0.107 and nearly tripped the 0.10 threshold gate. parse_resolved now
  prefers submitted_instances over total_instances (identical for
  full-split runs).

- 6/50 instances hit LimitsExceeded after 75 steps and submitted NOTHING,
  despite forensics showing fixes can be complete mid-run. patched
  process_instance now falls back to submitting `git diff` of the working
  tree when an instance ends abnormally with a live sandbox (requires rc 0
  and a `diff --git` prefix so an error string can never become a patch).
  Empty submissions score zero, so the fallback is strictly >=.

- Stage the swebench harness report as swebench_report_<task>.json and
  upload it; it names resolved/unresolved per instance and was previously
  left behind on the node.

* fix(swebench): hook budget-exhaustion fallback on the normal-return path

Run-2/3 verified the sandbox-cleanup patches (applied on the node, sweep
found 0 lingering sandboxes) but 0 fallback submissions fired while 6
instances still ended LimitsExceeded with empty patches. Root cause: mini's
agent run loop absorbs InterruptAgentFlow (Submitted, LimitsExceeded, ...)
and RETURNS normally with an empty submission -- LimitsExceeded never
reaches process_instance's except branch, which is where the fallback hook
lived (their trajectories carry no traceback/exception_str keys,
confirming the normal-return path).

Move the primary hook to just after agent.run(): any empty submission with
a live sandbox now submits `git diff` of the tree (same rc-0 +
"diff --git"-prefix guards). The except-path hook stays for real
exceptions.

* fix(swebench): cut scoring cost ~2x by patching eval-sandbox cpu=4 -> 2

Two full-300 Modal scorings measured ~$80 each in eval sandboxes alone
(vs $0.99-5.91 for image builds -- caching was never the cost driver).
Root cause: swebench's run_evaluation_modal.py hardcodes cpu=4 per
sandbox; Modal bills reserved cores and the test runs are predominantly
single-threaded pytest.

Patch the installed file at dep install (idempotent, anchor-checked,
numeric-validated) to SWEBENCH_EVAL_SANDBOX_CPU (default 2). Per-instance
tests run somewhat slower on fewer cores; scoring parallelism absorbs it.

* fix(swebench): completion watchdog + partial-preds salvage for generation

Run 29039988325 (full-300, workers=144): all 300 predictions were on disk
by t+60min but mini-extra never exited -- a probabilistic hang-on-exit at
high worker counts (the identical previous run exited cleanly). The
process idled 3h into SWEBENCH_AGENT_TIMEOUT, and the rc!=0 path then
deleted the complete preds.json.

- Completion watchdog: run mini-extra in the background and poll
  preds.json (written incrementally per instance); once all expected
  instances are present, grant SWEBENCH_AGENT_EXIT_GRACE (300s) for a
  clean exit, then kill and count generation complete. Overall
  SWEBENCH_AGENT_TIMEOUT deadline retained.
- Salvage: if generation fails with N>0 predictions written, warn and
  score the partial set instead of discarding real work (denominator is
  submitted instances, so partial runs report honestly over what ran).
- Tests: hung-mini watchdog kill, partial-preds salvage, zero-preds
  still-fails.

* feat(swebench): production defaults -- 0.50 threshold, 50-slice CI default, workers=64

Decisions 2026-07-09 after three full-300 validation runs (162/162/163
resolved, 54.0-54.3%):

- Threshold 0.10 -> 0.50: the old value predates the denominator fix and
  was effectively decorative; 0.50 sits ~4pts under the observed full-run
  floor and well under the 50-slice range (62-68%).
- EVAL_LIMIT empty now defaults to the 50-instance CI slice (~45min GPU +
  ~$9 Modal); EVAL_LIMIT=full runs the whole split (~1.75h + ~$44) for
  release-grade checks. Applies to the agentic swebench path only.
- SWEBENCH_AGENT_WORKERS default 8 -> 64: saturates the serving point
  without entering the 144-worker teardown-race territory; full-300
  generation drops ~3h -> ~55min.
- Known-bad b300 nodes (005: NCCL init death, 006: wedged nvidia driver)
  excluded via SALLOC_EXCLUDE with a tracking comment; remove as infra
  repairs them.

* fix(swebench): scenario implies generation mode when SWEBENCH_GEN_MODE unset

Label/changelog-triggered evals pass no swebench-gen-mode, which fell
through to single-shot generation -- ~10% resolved on a healthy config,
an instant false-negative against the new 0.50 gate. Agentic scenarios
now default to the agent loop; explicit SWEBENCH_GEN_MODE still wins.

* feat(swebench): agentic-only generation

Unset SWEBENCH_GEN_MODE now means the agent loop unconditionally, not
just for agentic scenarios -- SWE-bench without the agent loop is not a
meaningful eval (~10% resolved) and the 0.50 gate is calibrated to
agentic scores. single-shot remains solely as an explicit
SWEBENCH_GEN_MODE=single-shot debugging escape hatch.

* fix(swebench): launch-line step_limit display default matches the real default (75)

* fix(swebench): review-wave hardening -- 17 confirmed findings across the eval surface

From a 6-dimension review (bash correctness, python/patch code, workflow
wiring, docs-vs-behavior, credential hygiene, failure modes) with
adversarial verification (41 raw -> 17 confirmed, 24 refuted):

- SWEBENCH_USE_MODAL=false no longer passes --modal (the :+ expansion
  fired on any non-empty value, including "false")
- EVAL_LIMIT is validated in the agentic path: positive integer, "full",
  or 0 -- a negative/garbage value silently short-circuited the
  completion watchdog
- fail-fast when the task YAML dataset_path is not SWE-bench_Lite:
  agentic generation is hardcoded to the lite subset, and a divergent
  scoring dataset would mis-score every instance
- set -u safety: bare ${RESULT_FILENAME} and ${SPEC_DECODING} in
  append_lm_eval_summary
- explicit UTF-8 (reads tolerant, writes clean) on all swebench_score.py
  file I/O
- sandbox-sweep scope documented precisely: confined to the current Modal
  environment; concurrent agentic-eval legs need per-leg MODAL_ENVIRONMENT
  (workflow follow-up) before the matrix ever fans out in parallel
- docs/comments made truthful: knob defaults (workers 64, step limit 75),
  gen-mode input descriptions (empty = agentic), stale 0.10-threshold and
  dev-Mac-only framing, dangling _patch_swebench_agent_cleanup reference

* fix(swebench): terminate eval sandboxes on instance completion (scoring utilization)

run_instance_modal never finalizes its ModalSandboxRuntime -- the __exit__
that terminates the sandbox exists but nothing calls it -- so every eval
sandbox idle-bills after its tests finish until the 30-min sandbox timeout
or ephemeral-app teardown. Invisible on the 50-slice (all-fast tests, the
app ends in ~2 min and reaps everything); on full-300 the slow tail keeps
the app alive ~40 min and all 300 sandboxes bill ~30 min for ~3 min of
work: measured 152 sandbox-hours ($41.57 at cpu=2) where real test time is
~15-20 sandbox-hours.

Patch (same install-time mechanism, idempotent, anchor-checked): a
finally: on run_instance_modal's main try/except chain terminates the
sandbox on every exit path. Expected full-300 scoring: ~$41 -> ~$5-8.
_patch_swebench_scoring_cpu renamed _patch_swebench_scoring (cpu +
lifecycle hunks).

* feat(swebench): SWEBENCH_AGENT_SANDBOX_CPU knob for agent execution sandboxes

Modal's default sandbox reservation is fractional-core, and the agents run
real test suites inside these sandboxes (verify-before-submit), where a
starved CPU can eat the 300s command timeout and waste agent steps. Optional
knob threads through mini's modal_sandbox_kwargs -> swe-rex ->
modal.Sandbox.create; unset preserves the Modal default (current behavior).
Added for the cost/time pareto sweep.

* feat(swebench): SWEBENCH_EVAL_TIMEOUT knob for per-instance scoring timeout

Post-lifecycle-patch, real test runs bill ~33s each and the scoring wall
(~29 min on full-300) is set almost entirely by the 7 persistently-erroring
instances running to the harness's 1800s default. Optional pass-through to
run_evaluation --timeout; unset preserves the harness default.

* feat(swebench): conc-matched agent workers, 900s scoring timeout, app rename, app-scoped sweep

Pareto-sweep conclusions (11 runs):

- SWEBENCH_AGENT_WORKERS defaults to the config's CONC (else 64): the eval
  drives the server at the concurrency its config was tuned for. At conc144
  this cut full-run generation 90m -> 51m at identical score (five full
  runs: 160-163/300); the old w144 hang risk is contained by the completion
  watchdog (three clean w144 runs since).
- SWEBENCH_EVAL_TIMEOUT defaults to 900s: real test runs bill ~33s; only
  the persistently-erroring instances touch the ceiling and they gate the
  scoring tail.
- swe-rex's hardcoded Modal app name is patched to SWEBENCH_MODAL_APP_NAME
  (default infx-evals-swe) so the dashboard shows ours, not the library's.
- The post-generation sweep is now scoped to that app via
  Sandbox.list(app_id=...) -- it can no longer touch other apps' sandboxes
  in the shared workspace (narrows the concurrent-tenant hazard to
  same-app legs only).

Agent-sandbox CPU knob stays unset by default: the sweep measured ~zero
command timeouts at Modal's fractional-core default (1 in 300 on the full
set) -- the agent loop is inference-bound and boosting is pure cost.

* docs: waiver for eval-harness runtime patches (Check 9, PR #1947)

The SWE-bench eval patches three pinned eval-tooling packages at install
time (mini-swe-agent, swe-rex, swebench harness) -- sandbox-lifecycle and
cost fixes measured at ~17x Modal spend reduction. No inference engine or
serving stack is touched; the waiver is filed proactively because the
mechanical shape (heredoc patches in benchmark_lib.sh) matches Check 9's
inline-patch pattern.

* docs: drop eval-tooling waiver -- maintainer call: Check 9 targets engine/serving patches (vendor patchwork), not our own eval-harness tooling; vLLM runs as shipped throughout

* fix(swebench): gate the one un-gated agentic recipe; pin swebench; refresh EVALS.md

Merge-readiness sweep findings:

- BLOCKER: dsv4_fp4_mi355x_vllm.sh arrived via a main-merge (#2109) after the
  eval-gating rollout, so it was the only 1 of 23 single-node agentic recipes
  without the EVAL_ONLY/maybe_run_eval block. A live config targets it
  (configs/amd-master.yaml: dsv4-fp4-mi355x-vllm-agentic), which the generator
  marks for eval -- under EVAL_ONLY=true the recipe would fall through to the
  throughput replay and produce no score (failing the eval-scores gate).
  Appended the standard gating block (matches its sglang sibling).
- swebench install pinned to ==4.1.0: the harness CLI flags and the
  _patch_swebench_scoring anchors are verified against 4.1.0; an unpinned
  upgrade could drift either.
- EVALS.md refreshed to the shipped behavior: agentic-only default, 0.50 gate,
  50-slice/full run sizing, and the current knob set/defaults.

* docs+test: clear merge-readiness-sweep nits (docs truth + EVAL_LIMIT guard test)

Second readiness sweep returned GO (no blockers; mi355x gate fix held).
Clearing the actionable nits:

- EVALS.md: --all-evals no longer says agentic configs are excluded (they're
  included and run swebench under evals-only/all-evals; excluded only from the
  default non-eval sweep).
- generate_sweep_configs.py: three "(single-shot)" comments corrected to
  agentic (generation is agentic-only).
- benchmark-tmpl.yml: eval-limit input description corrected (empty = 50-slice
  swebench default, not "full set").
- tests: cover the EVAL_LIMIT positive-integer rejection guard (-5/abc/3.5 ->
  fail fast) and the full/0 whole-split sentinels. 53 eval tests pass.

modal left unpinned deliberately: unlike the three source-patched packages, it
is a client to a live hosted service where an exact pin invites client/server
skew.

* refactor(evals): move inline runtime patches to utils/evals/patches/

The lm-eval sitecustomize, mini-swe-agent/swe-rex, and swebench Modal
scorer patches were embedded in benchmark_lib.sh as heredocs. Move each
verbatim into a standalone Python file under utils/evals/patches/ (with
the rationale comments as docstrings) and have the _patch_* shell
helpers invoke them via a BASH_SOURCE-anchored path, matching how
run_lm_eval already anchors task YAMLs.

Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(evals): convert thresholds.json to thresholds.yaml

The repo uses YAML for configuration everywhere else, so move the eval
thresholds config to YAML too. validate_scores.py now parses the config
with yaml.safe_load (JSON is a YAML subset, so legacy JSON configs via
--thresholds still load); on runner hosts without PyYAML, JSON configs
fall back to the stdlib json module and YAML configs fail with an
actionable error instead of silently weakening the gate.

Requested by @cquil11 in PR #1947 review.

Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(evals): clean runtime patch integration

* style(evals): trim implementation comments

* style(evals): tighten rationale comments

* refactor(evals): call unified eval entrypoint

* docs(evals): explain lm-eval task arguments

* fix(evals): run full SWE-bench Lite by default

* fix(evals): schedule AgentX evals from changelog

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: 4 people

.github/workflows/benchmark-tmpl.yml

@@ -116,6 +116,16 @@ on:
         required: false
         type: string
         default: '3600'
+      eval-limit:
+        description: "Eval instance count: empty/full = whole split (default); N = first-N smoke slice"
+        required: false
+        type: string
+        default: ""
+      swebench-gen-mode:
+        description: "SWE-bench generation mode (single-shot | agentic). Empty = agentic (single-shot is an explicit debugging escape hatch)."
+        required: false
+        type: string
+        default: ""
 env:
   RANDOM_RANGE_RATIO: 0.8
   HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }}
@@ -151,10 +161,18 @@ env:
   KV_P2P_TRANSFER: ${{ inputs.kv-p2p-transfer }}
   TOTAL_CPU_DRAM_GB: ${{ inputs.total-cpu-dram-gb }}
   DURATION: ${{ inputs.duration }}
+  EVAL_LIMIT: ${{ inputs.eval-limit }}
+  SWEBENCH_GEN_MODE: ${{ inputs.swebench-gen-mode }}
   AIPERF_FAILED_REQUEST_THRESHOLD: '0.10'
   RESULT_DIR: /workspace/results
   PYTHONDONTWRITEBYTECODE: '1'
   PYTHONPYCACHEPREFIX: /tmp/inferencex-pycache
+  # GPU runners lack Docker for SWE-bench scoring.
+  SWEBENCH_USE_MODAL: 'true'
+  MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
+  MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
+  # These b300 nodes are currently broken.
+  SALLOC_EXCLUDE: 'b300-005,b300-006'
 
 permissions:
   contents: read
@@ -311,6 +329,10 @@ jobs:
             meta_env.json
             results*.json
             sample*.jsonl
+            agent_preds.json
+            predictions.jsonl
+            swebench_report_*.json
+            *.traj*
           if-no-files-found: ${{ inputs.eval-only && 'error' || 'ignore' }}
 
       - name: Verify eval scores
@@ -324,6 +346,7 @@ jobs:
           # Remove any eval results JSONs that were moved into workspace
           rm -f results*.json || true
           rm -f sample*.jsonl || true
+          rm -f agent_preds.json predictions.jsonl swebench_report_*.json *.traj* || true
           
       - name: Resource cleanup (post-run)
         if: always()

.github/workflows/e2e-tests.yml

@@ -21,6 +21,16 @@ on:
                 required: false
                 type: string
                 default: ""
+            eval-limit:
+                description: "Eval instance count: empty/full = whole split (default); N = first-N smoke slice"
+                required: false
+                type: string
+                default: ""
+            swebench-gen-mode:
+                description: "SWE-bench generation mode (single-shot | agentic). Empty = agentic (single-shot is an explicit debugging escape hatch)."
+                required: false
+                type: string
+                default: ""
     workflow_call:
         inputs:
             generate-cli-command:
@@ -40,6 +50,16 @@ on:
                 required: false
                 type: string
                 default: ""
+            eval-limit:
+                description: "Eval instance count: empty/full = whole split (default); N = first-N smoke slice"
+                required: false
+                type: string
+                default: ""
+            swebench-gen-mode:
+                description: "SWE-bench generation mode (single-shot | agentic). Empty = agentic (single-shot is an explicit debugging escape hatch)."
+                required: false
+                type: string
+                default: ""
 
 jobs:
     get-jobs:
@@ -50,6 +70,7 @@ jobs:
             eval-config: ${{ steps.get-jobs.outputs.eval-config }}
             multi-node-eval-config: ${{ steps.get-jobs.outputs.multi-node-eval-config }}
             agentic-config: ${{ steps.get-jobs.outputs.agentic-config }}
+            agentic-eval-config: ${{ steps.get-jobs.outputs.agentic-eval-config }}
             multi-node-agentic-config: ${{ steps.get-jobs.outputs.multi-node-agentic-config }}
         steps:
             - name: Checkout code (ref)
@@ -69,13 +90,15 @@ jobs:
                   pip install pydantic
                   CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix_logic/generate_sweep_configs.py \
                     ${{ inputs.generate-cli-command || github.event.inputs.generate-cli-command }})
-                  AGENTIC=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' not in x]))")
+                  AGENTIC=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' not in x and not x.get('run-eval', False)]))")
+                  AGENTIC_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' not in x and x.get('run-eval', False)]))")
                   MULTI_AGENTIC=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' in x]))")
                   SINGLE=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') != 'agentic-coding' and not x.get('eval-only', False)]))")
                   MULTI=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' in x and x.get('scenario-type') != 'agentic-coding' and not x.get('eval-only', False)]))")
                   EVALS=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') != 'agentic-coding' and x.get('run-eval', False)]))")
                   MULTI_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' in x and x.get('run-eval', False)]))")
                   echo "agentic-config=$AGENTIC" >> $GITHUB_OUTPUT
+                  echo "agentic-eval-config=$AGENTIC_EVAL" >> $GITHUB_OUTPUT
                   echo "multi-node-agentic-config=$MULTI_AGENTIC" >> $GITHUB_OUTPUT
                   echo "single-node-config=$SINGLE" >> $GITHUB_OUTPUT
                   echo "multi-node-config=$MULTI" >> $GITHUB_OUTPUT
@@ -223,6 +246,44 @@ jobs:
             scenario-type: agentic-coding
             ref: ${{ inputs.ref }}
 
+    test-sweep-agentic-evals:
+        needs: get-jobs
+        if: ${{ needs.get-jobs.outputs.agentic-eval-config != '[]' }}
+        uses: ./.github/workflows/benchmark-tmpl.yml
+        name: agentic eval /
+        strategy:
+            fail-fast: false
+            matrix:
+                config: ${{ fromJson(needs.get-jobs.outputs.agentic-eval-config) }}
+        secrets: inherit
+        with:
+            exp-name: ${{ matrix.config.exp-name }}
+            runner: ${{ matrix.config.runner }}
+            image: ${{ matrix.config.image }}
+            model: ${{ matrix.config.model }}
+            model-prefix: ${{ matrix.config.model-prefix }}
+            framework: ${{ matrix.config.framework }}
+            precision: ${{ matrix.config.precision }}
+            tp: ${{ matrix.config.tp }}
+            ep: ${{ matrix.config.ep }}
+            dp-attn: ${{ matrix.config.dp-attn }}
+            conc: ${{ matrix.config.conc }}
+            kv-offloading: ${{ matrix.config.kv-offloading }}
+            kv-offload-backend: ${{ matrix.config.kv-offload-backend }}
+            total-cpu-dram-gb: ${{ matrix.config.total-cpu-dram-gb }}
+            duration: ${{ inputs.duration-override != '' && inputs.duration-override || matrix.config.duration }}
+            isl: '0'
+            osl: '0'
+            max-model-len: '0'
+            spec-decoding: 'none'
+            disagg: 'false'
+            run-eval: true
+            eval-only: true
+            eval-limit: ${{ inputs.eval-limit }}
+            swebench-gen-mode: ${{ inputs.swebench-gen-mode }}
+            scenario-type: agentic-coding
+            ref: ${{ inputs.ref }}
+
     test-sweep-multi-node-agentic:
         needs: get-jobs
         if: ${{ needs.get-jobs.outputs.multi-node-agentic-config != '[]' }}
@@ -345,6 +406,7 @@ jobs:
             disagg: ${{ matrix.config.disagg }}
             run-eval: true
             eval-only: true
+            eval-limit: ${{ inputs.eval-limit }}
             ref: ${{ inputs.ref }}
 
     collect-results:
@@ -356,8 +418,8 @@ jobs:
             result-prefix: "bmk"
 
     collect-evals:
-        needs: [test-sweep-evals, test-sweep-multi-node-evals]
-        if: ${{ always() && (needs.test-sweep-evals.result != 'skipped' || needs.test-sweep-multi-node-evals.result != 'skipped') }}
+        needs: [test-sweep-evals, test-sweep-multi-node-evals, test-sweep-agentic-evals]
+        if: ${{ always() && (needs.test-sweep-evals.result != 'skipped' || needs.test-sweep-multi-node-evals.result != 'skipped' || needs.test-sweep-agentic-evals.result != 'skipped') }}
         uses: ./.github/workflows/collect-evals.yml
         secrets: inherit