Track effort per arm and report unfinished runs as unfinished

A leaderboard that shows only a score answers half the question. Wall time,
reported cost, and cost per resolved task are what both reference benchmarks
lead with, and they were missing or wrong here: Codex time and tokens were never
read at all, so an arm that spent twenty minutes of Codex effort showed a blank.
Both are now taken from the log each phase already writes.

An unfinished run was the worse problem. Scoring an arm that has run 21 of 50
tasks against 50 reports it as a 42% failure when it has actually solved
everything it reached. An arm is now scored against the tasks it ran, labelled
in progress, and the page says plainly that arms which have run different
numbers of tasks cannot be compared to each other.

A cost figure that covers only an arm's Claude phases is marked, because Codex,
GLM and Kimi report no cost and rendering that as $0.00 reads as free rather
than as unmeasured.

Evaluator reports now carry a run label. Named only by timestamp, reports from
two runs of one condition were indistinguishable and a page built across them
silently mixed a 50-task subset with a 1-task one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: alanshurafa

benchmarks/code/scripts/evaluate-swebench.sh

@@ -43,10 +43,23 @@ case "$MODE" in
   predictions)
     predictions="${1:-}"
     [[ -n "$predictions" ]] || { code_die "predictions mode needs a JSONL file"; exit 2; }
+    shift || true
+    # The evaluator names its report after the run id and nothing else, so two
+    # benchmark runs of the same condition are indistinguishable afterwards and
+    # a page built from them silently mixes subsets. --label stamps the batch
+    # into the run id so a report can be traced back to the run that made it.
+    LABEL="code-bench"
+    while (( $# > 0 )); do
+      case "$1" in
+        --label) LABEL="${2:?--label needs a value}"; shift 2 ;;
+        *) code_die "unknown evaluate option: $1"; exit 2 ;;
+      esac
+    done
+    [[ "$LABEL" =~ ^[A-Za-z0-9._-]+$ ]] || { code_die "--label must be filesystem-safe"; exit 2; }
     bash "$CODE_DIR/validate-predictions.sh" "$predictions" "$SUITE"
     predictions_dir=$(cd "$(dirname "$predictions")" && pwd -P)
     predictions="$predictions_dir/$(basename "$predictions")"
-    run_id="code-bench-$(date -u +%Y%m%dT%H%M%SZ)"
+    run_id="$LABEL-$(date -u +%Y%m%dT%H%M%SZ)"
     (cd "$EVAL_ROOT" && "$CLI" eval verified -p "$predictions" --run-id "$run_id" -j "${CODE_BENCH_EVAL_JOBS:-1}")
     ;;
   *)

benchmarks/site/aggregate.sh

@@ -10,11 +10,13 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
 RESULTS_ROOT="${CODE_BENCH_RESULTS_ROOT:-$REPO_ROOT/benchmarks/results/code}"
 SUITE="swebench-verified-canary"
+RUN_LABEL=""
 OUTPUT="$RESULTS_ROOT/site/leaderboard.json"
 
 while (( $# > 0 )); do
   case "$1" in
     --suite) SUITE="${2:?--suite needs a value}"; shift 2 ;;
+    --run-label) RUN_LABEL="${2:?--run-label needs a value}"; shift 2 ;;
     --output) OUTPUT="${2:?--output needs a value}"; shift 2 ;;
     *) printf 'ERROR: unknown aggregate option: %s\n' "$1" >&2; exit 2 ;;
   esac
@@ -23,11 +25,15 @@ done
 command -v python >/dev/null 2>&1 || { printf 'ERROR: python is required\n' >&2; exit 1; }
 [[ -d "$RESULTS_ROOT/evaluation" ]] || { printf 'ERROR: no evaluation directory under %s\n' "$RESULTS_ROOT" >&2; exit 1; }
 
+label_args=()
+[[ -n "$RUN_LABEL" ]] && label_args=(--run-label "$RUN_LABEL")
+
 python "$SCRIPT_DIR/build-site-data.py" \
   --repo-root "$REPO_ROOT" \
   --results-root "$RESULTS_ROOT" \
   --suite "$SUITE" \
   --output "$OUTPUT" \
+  ${label_args[@]+"${label_args[@]}"} \
   --generated-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 
 python "$SCRIPT_DIR/render-page.py" --data "$OUTPUT" --output "${OUTPUT%%.json}.html"

benchmarks/site/build-site-data.py

@@ -53,14 +53,22 @@ def harness_dirty(root):
     return bool(proc.stdout.strip()) if proc.returncode == 0 else None
 
 
-def newest_reports(eval_dir):
-    """Latest evaluator report per condition, plus the ones it supersedes."""
+def newest_reports(eval_dir, run_label=None):
+    """Latest evaluator report per condition, plus the ones it supersedes.
+
+    A report is named only for the evaluator run that produced it, so reports
+    from two benchmark runs of the same condition are indistinguishable and a
+    page built across them silently mixes subsets. run_label restricts the scan
+    to one batch, which is what makes a per-run page possible.
+    """
     latest, superseded = {}, []
     for path in sorted(glob.glob(os.path.join(eval_dir, '*.json'))):
         match = REPORT_NAME_RE.match(os.path.basename(path))
         if not match:
             continue
         cond, run_id = match.group('cond'), match.group('run_id')
+        if run_label is not None and not run_id.startswith(run_label + '-'):
+            continue
         previous = latest.get(cond)
         if previous is None or run_id > previous[0]:
             if previous is not None:
@@ -184,6 +192,8 @@ def cell_telemetry(cell_dir):
         'claude_output_tokens': 0,
         'claude_wall_seconds': 0,
         'codex_phases': 0,
+        'codex_wall_seconds': 0,
+        'codex_tokens': 0,
         'single_shot_attempts': None,
         'sandbox': None,
     }
@@ -220,9 +230,55 @@ def cell_telemetry(cell_dir):
                 # Each Codex phase writes both a transcript and a stderr log.
                 # Counting every .log double-counted every phase.
                 out['codex_phases'] += 1
+            elif name.startswith('codex-') and name.endswith('.stderr.log'):
+                seconds, tokens = codex_effort(path)
+                out['codex_wall_seconds'] += seconds
+                out['codex_tokens'] += tokens
     return out
 
 
+CODEX_TS_RE = re.compile(r'^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.\d+)?Z')
+CODEX_TOKENS_RE = re.compile(r'^([\d,]+)$')
+
+
+def codex_effort(stderr_path):
+    """Wall seconds and token count for one Codex phase, from its own log.
+
+    The Codex CLI reports no cost, so tokens and elapsed time are the only
+    honest units of effort available for it. Both are read out of the log the
+    phase already wrote: the span between its first and last timestamp, and the
+    figure it prints under "tokens used".
+    """
+    first = last = None
+    tokens = 0
+    want_tokens = False
+    try:
+        with open(stderr_path, encoding='utf-8', errors='replace') as handle:
+            for line in handle:
+                match = CODEX_TS_RE.match(line)
+                if match:
+                    if first is None:
+                        first = match.group(1)
+                    last = match.group(1)
+                stripped = line.strip()
+                if want_tokens:
+                    count = CODEX_TOKENS_RE.match(stripped)
+                    if count:
+                        tokens += int(count.group(1).replace(',', ''))
+                    want_tokens = False
+                elif stripped == 'tokens used':
+                    want_tokens = True
+    except OSError:
+        return 0, 0
+    seconds = 0
+    if first and last and last >= first:
+        import datetime
+        fmt = '%Y-%m-%dT%H:%M:%S'
+        seconds = int((datetime.datetime.strptime(last, fmt)
+                       - datetime.datetime.strptime(first, fmt)).total_seconds())
+    return seconds, tokens
+
+
 def main():
     ap = argparse.ArgumentParser()
     ap.add_argument('--repo-root', required=True)
@@ -231,6 +287,8 @@ def main():
     ap.add_argument('--output', required=True)
     ap.add_argument('--generated-at', required=True,
                     help='UTC timestamp supplied by the caller')
+    ap.add_argument('--run-label', default=None,
+                    help='only read evaluator reports from this labelled batch')
     args = ap.parse_args()
 
     root = os.path.abspath(args.repo_root)
@@ -250,7 +308,7 @@ def main():
     repos = {row['instance_id']: row['repo'] for row in subset['instances']}
     lock = read_json(os.path.join(code_dir, 'external-sources.lock.json'))
 
-    latest, superseded = newest_reports(eval_dir)
+    latest, superseded = newest_reports(eval_dir, args.run_label)
     cells = index_cells(runs_root)
     attempts_index = index_attempts(runs_root)
 
@@ -276,6 +334,8 @@ def main():
                 'claude_output_tokens': 0,
                 'claude_wall_seconds': 0,
                 'codex_phases': 0,
+                'codex_wall_seconds': 0,
+                'codex_tokens': 0,
                 'cells_linked': 0,
                 'sandbox_modes': [],
                 'single_shot_attempts': [],
@@ -334,6 +394,8 @@ def main():
                     row['telemetry']['claude_output_tokens'] += telemetry['claude_output_tokens']
                     row['telemetry']['claude_wall_seconds'] += telemetry['claude_wall_seconds']
                     row['telemetry']['codex_phases'] += telemetry['codex_phases']
+                    row['telemetry']['codex_wall_seconds'] += telemetry['codex_wall_seconds']
+                    row['telemetry']['codex_tokens'] += telemetry['codex_tokens']
                     if telemetry['sandbox']:
                         sandboxes.add(telemetry['sandbox'])
                     if telemetry['single_shot_attempts'] is not None:
@@ -354,6 +416,35 @@ def main():
                                  if attempt else None),
                 })
         row['telemetry']['claude_cost_usd'] = round(row['telemetry']['claude_cost_usd'], 4)
+
+        # A task the arm actually ran, whether or not it yielded a patch. The
+        # distinction matters: a task that ran and produced nothing is a zero,
+        # but a task that was never reached is not a result at all. Scoring an
+        # unreached task as zero understates an interrupted run as badly as
+        # dropping a failed one would flatter a finished one.
+        ran = [task for task in row['per_task']
+               if task['status'] in ('resolved', 'unresolved', 'no-patch')]
+        row['attempted_count'] = len(ran)
+        row['complete'] = len(ran) == len(instances)
+
+        # Effort per arm. Codex, GLM and Kimi report no cost, so an arm that
+        # uses them has a dollar figure covering only its Claude phases; saying
+        # so is the difference between a partial figure and a wrong one.
+        telemetry = row['telemetry']
+        telemetry['total_wall_seconds'] = (telemetry['claude_wall_seconds']
+                                           + telemetry['codex_wall_seconds'])
+        telemetry['cost_is_complete'] = (
+            telemetry['codex_phases'] == 0
+            and not telemetry['single_shot_attempts'])
+        resolved = row['resolved'] or 0
+        # No reported cost is not zero cost: Codex, GLM and Kimi bill elsewhere.
+        # Emitting 0.0 here would render as "$0.00 per resolved task", which
+        # reads as free rather than as unmeasured.
+        telemetry['cost_per_resolved'] = (
+            round(telemetry['claude_cost_usd'] / resolved, 4)
+            if resolved and telemetry['claude_cost_usd'] else None)
+        telemetry['seconds_per_resolved'] = (
+            round(telemetry['total_wall_seconds'] / resolved) if resolved else None)
         rows.append(row)
 
     payload = {