Merge pull request #165 from vercel-labs/jude/agent-eval-eval-staleness

[agent-eval] Incremental eval staleness: status / run + content-aware fingerprints

Author: gaojude

.changeset/eval-staleness.md

@@ -0,0 +1,11 @@
+---
+"@vercel/agent-eval": minor
+---
+
+Incremental eval-staleness workflow, so adopting a changed or new eval doesn't force re-running every experiment.
+
+- Fingerprint split: each result stores a content-only hash next to the combined (content+config) one. A real eval change is never masked; a benign config change (e.g. a `timeout` bump, or pinning a judge) is carried forward by `refingerprint` instead of re-running. Existing `fingerprint` values are byte-identical, so caches stay valid. (Fixes the previous re-fingerprinting that silently re-stamped every result and hid eval changes.)
+- `agent-eval status` — read-only: which evals are new vs changed, per experiment (classified by content). `--check` exits non-zero on any new/changed eval (a simple CI gate); `--json` emits per-experiment new/changed so a consumer can apply its own "which staleness is acceptable" policy.
+- `agent-eval run <experiments...>` — run the named experiments' new/changed evals (auto-carries config-only changes first).
+- Bare `agent-eval` shows status, then (in a terminal) lets you multi-select which experiments to run — it never re-runs everything.
+- Removes `run-all` and `--dry` (the run-everything-when-stale behavior). There is no in-framework "acknowledge/keep" — staleness acceptance is the consumer's policy (e.g. filter `status --json` against an accepted-stale list in CI).

README.md

@@ -541,6 +541,54 @@ On subsequent runs, evals with a matching fingerprint and a valid cached result
 
 Use `--force` to bypass fingerprinting and re-run everything. Functions like `setup` and `editPrompt` cannot be hashed, so use `--force` when you change those.
 
+Each result also stores a `contentFingerprint` — a hash of the eval files **only**, independent of config. This separates "the eval itself changed" from "a config field changed."
+
+### Carrying forward config-only changes
+
+A benign config change (e.g. bumping `timeout`) changes the combined fingerprint and would otherwise re-run every eval. `agent-eval refingerprint` carries those forward in the cached results **without masking a real eval change**:
+
+```bash
+agent-eval refingerprint            # all experiments
+agent-eval refingerprint cc --dry   # preview one experiment
+```
+
+For each cached result it compares the eval's current `contentFingerprint` to the stored one: if the content is unchanged it re-stamps the combined fingerprint (the result stays cached); if the content **changed** it leaves the result stale so it re-runs. `agent-eval status` already classifies by eval *content*, so it never reports a config-only change as work — run `refingerprint` after editing an experiment config to carry that change into the cache (`run` does this automatically).
+
+### After changing or syncing evals: status → pick what to run
+
+Run `agent-eval` with no arguments. It shows the work, then — in a terminal — lets you multi-select which experiments to run. It never re-runs everything:
+
+```bash
+agent-eval
+```
+```
+Evals needing work:
+  new      agent-026-no-serial-await
+  changed  agent-024-avoid-redundant-usestate
+
+Work to do — 6 run(s) across 3 experiment(s):
+  claude-opus-4.6      2 to run  (22 up to date)
+  ...
+
+Pick experiments to run:
+   1  claude-opus-4.6
+   2  claude-sonnet-4.6
+Numbers (e.g. 1,3), "all", or Enter to skip:
+```
+
+Status classifies each eval by **content**, so a benign config change (e.g. pinning a judge) is never reported as work. The same building blocks work non-interactively:
+
+```bash
+agent-eval status                  # read-only: what's new/changed, per experiment
+agent-eval status --check          # exit non-zero if anything is new/changed (simple CI gate)
+agent-eval status --json           # machine-readable, for custom CI policy
+agent-eval run claude-sonnet-4.6   # run the named experiment(s) — new/changed evals only
+```
+
+**Accepting staleness is the consumer's call, not the framework's.** `agent-eval` only *reports* — it has no `keep`/`acknowledge`. If you want to leave some experiments on an older eval while keeping others fresh, do that in your own CI: read `agent-eval status --json` (per-experiment `new`/`changed`) and fail only on experiments not in your accepted-stale list. (See next-evals-oss's `scripts/check-stale.mjs` for an example.)
+
+> `refingerprint` (carry config-only changes forward) runs automatically inside `run`; your sync script should call `agent-eval refingerprint` after pulling evals so committed results pick up benign config changes without re-running.
+
 ## Failure Classification
 
 When evals fail, the framework optionally classifies each failure as one of:

packages/agent-eval/src/cli.test.ts

@@ -1,9 +1,11 @@
 import { describe, it, expect, beforeEach, afterEach } from 'vitest';
 import { execSync } from 'child_process';
-import { mkdirSync, writeFileSync, rmSync, existsSync } from 'fs';
+import { mkdirSync, writeFileSync, readFileSync, rmSync, existsSync } from 'fs';
 import { join, resolve } from 'path';
 import { fileURLToPath } from 'url';
 import { dirname } from 'path';
+import { computeContentFingerprint, computeFingerprint } from './lib/fingerprint.js';
+import { loadConfig } from './lib/config.js';
 
 const __filename = fileURLToPath(import.meta.url);
 const __dirname = dirname(__filename);
@@ -51,9 +53,9 @@ describe('CLI', () => {
   });
 
   describe('run command', () => {
-    it('shows error when config file does not exist', () => {
-      const result = runCli(['run', '/non/existent/config.ts']);
-      expect(result.stderr).toContain('not found');
+    it('shows error when the named experiment does not exist', () => {
+      const result = runCli(['run', 'no-such-experiment']);
+      expect(result.stderr.toLowerCase()).toMatch(/no experiments matched|required/);
       expect(result.exitCode).toBe(1);
     });
 
@@ -170,4 +172,164 @@ describe('CLI', () => {
       expect(result.exitCode).toBe(1);
     });
   });
+
+  describe('refingerprint command', () => {
+    function setupProject(): { projectDir: string; evalPath: string; summaryPath: string } {
+      const projectDir = join(TEST_DIR, 'refp');
+      const experimentsDir = join(projectDir, 'experiments');
+      mkdirSync(experimentsDir, { recursive: true });
+      writeFileSync(join(experimentsDir, 'cc.ts'), `export default { agent: 'claude-code', model: 'opus' };`);
+      const evalDir = join(projectDir, 'evals', 'eval-1');
+      mkdirSync(evalDir, { recursive: true });
+      writeFileSync(join(evalDir, 'PROMPT.md'), 'do it');
+      writeFileSync(join(evalDir, 'EVAL.ts'), 'test code');
+      writeFileSync(join(evalDir, 'package.json'), '{"type":"module"}');
+      const summaryDir = join(projectDir, 'results', 'cc', '2026-01-01T00-00-00.000Z', 'eval-1');
+      mkdirSync(summaryDir, { recursive: true });
+      return { projectDir, evalPath: evalDir, summaryPath: join(summaryDir, 'summary.json') };
+    }
+
+    it('carries forward a config-only change but never masks a content change', () => {
+      const { projectDir, evalPath, summaryPath } = setupProject();
+      const currentContent = computeContentFingerprint(evalPath);
+
+      // Config-only change: stored content fp matches current, combined is stale.
+      writeFileSync(
+        summaryPath,
+        JSON.stringify({
+          totalRuns: 1, passedRuns: 1, passRate: '100%', meanDuration: 1,
+          fingerprint: 'STALE_COMBINED', contentFingerprint: currentContent,
+        })
+      );
+      let r = runCli(['refingerprint'], projectDir);
+      expect(r.exitCode).toBe(0);
+      let summary = JSON.parse(readFileSync(summaryPath, 'utf-8'));
+      expect(summary.fingerprint).not.toBe('STALE_COMBINED'); // carried forward
+      expect(summary.fingerprint).toMatch(/^[a-f0-9]{64}$/);
+      expect(summary.contentFingerprint).toBe(currentContent); // content untouched
+
+      // Content change: stored content fp differs → must be left stale, NOT masked.
+      writeFileSync(
+        summaryPath,
+        JSON.stringify({
+          totalRuns: 1, passedRuns: 1, passRate: '100%', meanDuration: 1,
+          fingerprint: 'OLD_COMBINED', contentFingerprint: 'OLD_CONTENT',
+        })
+      );
+      r = runCli(['refingerprint'], projectDir);
+      expect(r.exitCode).toBe(0);
+      summary = JSON.parse(readFileSync(summaryPath, 'utf-8'));
+      expect(summary.fingerprint).toBe('OLD_COMBINED'); // NOT re-stamped
+      expect(summary.contentFingerprint).toBe('OLD_CONTENT');
+    });
+
+    it('--dry does not write', () => {
+      const { projectDir, evalPath, summaryPath } = setupProject();
+      const currentContent = computeContentFingerprint(evalPath);
+      writeFileSync(
+        summaryPath,
+        JSON.stringify({
+          totalRuns: 1, passedRuns: 1, passRate: '100%', meanDuration: 1,
+          fingerprint: 'STALE_COMBINED', contentFingerprint: currentContent,
+        })
+      );
+      const r = runCli(['refingerprint', '--dry'], projectDir);
+      expect(r.exitCode).toBe(0);
+      const summary = JSON.parse(readFileSync(summaryPath, 'utf-8'));
+      expect(summary.fingerprint).toBe('STALE_COMBINED'); // unchanged under --dry
+    });
+  });
+
+  describe('staleness flow (status / refingerprint / --check / --json)', () => {
+    it('fresh → change → status + --check + --json flag it; refingerprint stays honest; a rerun clears it', async () => {
+      const projectDir = join(TEST_DIR, 'flow');
+      const experimentsDir = join(projectDir, 'experiments');
+      mkdirSync(experimentsDir, { recursive: true });
+      writeFileSync(join(experimentsDir, 'cc.ts'), `export default { agent: 'claude-code', model: 'opus' };`);
+      const evalDir = join(projectDir, 'evals', 'eval-1');
+      mkdirSync(evalDir, { recursive: true });
+      writeFileSync(join(evalDir, 'PROMPT.md'), 'do it');
+      writeFileSync(join(evalDir, 'EVAL.ts'), 'v1');
+      writeFileSync(join(evalDir, 'package.json'), '{"type":"module"}');
+      const summaryPath = join(projectDir, 'results', 'cc', '2026-01-01T00-00-00.000Z', 'eval-1', 'summary.json');
+      mkdirSync(dirname(summaryPath), { recursive: true });
+
+      const config = await loadConfig(join(experimentsDir, 'cc.ts'));
+      const modelConfig = { ...config, model: Array.isArray(config.model) ? config.model[0] : config.model };
+      const seedFresh = () =>
+        writeFileSync(
+          summaryPath,
+          JSON.stringify({
+            totalRuns: 1, passedRuns: 1, passRate: '100%', meanDuration: 1,
+            fingerprint: computeFingerprint(evalDir, modelConfig as never),
+            contentFingerprint: computeContentFingerprint(evalDir),
+          })
+        );
+
+      // 1. Fresh → status clean, --check passes.
+      seedFresh();
+      expect(runCli(['status'], projectDir).stdout).toContain('up to date');
+      expect(runCli(['status', '--check'], projectDir).exitCode).toBe(0);
+
+      // 2. Eval content changes → status flags it, --check fails, --json reports it.
+      writeFileSync(join(evalDir, 'EVAL.ts'), 'v2');
+      const s = runCli(['status'], projectDir).stdout;
+      expect(s).toContain('changed');
+      expect(s).toContain('eval-1');
+      expect(runCli(['status', '--check'], projectDir).exitCode).toBe(1);
+      const json = JSON.parse(runCli(['status', '--json'], projectDir).stdout);
+      expect(json.work).toEqual([{ experiment: 'cc', new: [], changed: ['eval-1'] }]);
+
+      // 3. refingerprint must NOT mask a content change — still failing.
+      runCli(['refingerprint'], projectDir);
+      expect(runCli(['status', '--check'], projectDir).exitCode).toBe(1);
+
+      // 4. A rerun (simulated by re-seeding fresh for the new content) clears it.
+      seedFresh();
+      expect(runCli(['status', '--check'], projectDir).exitCode).toBe(0);
+    });
+
+    it('status reports new and changed evals as the work to do', async () => {
+      const projectDir = join(TEST_DIR, 'status');
+      const experimentsDir = join(projectDir, 'experiments');
+      mkdirSync(experimentsDir, { recursive: true });
+      writeFileSync(join(experimentsDir, 'cc.ts'), `export default { agent: 'claude-code', model: 'opus' };`);
+      const evalDir = join(projectDir, 'evals', 'eval-1');
+      mkdirSync(evalDir, { recursive: true });
+      writeFileSync(join(evalDir, 'PROMPT.md'), 'do it');
+      writeFileSync(join(evalDir, 'EVAL.ts'), 'v1');
+      writeFileSync(join(evalDir, 'package.json'), '{"type":"module"}');
+
+      const config = await loadConfig(join(experimentsDir, 'cc.ts'));
+      const modelConfig = { ...config, model: Array.isArray(config.model) ? config.model[0] : config.model };
+      const sp = join(projectDir, 'results', 'cc', '2026-01-01T00-00-00.000Z', 'eval-1', 'summary.json');
+      mkdirSync(dirname(sp), { recursive: true });
+      writeFileSync(
+        sp,
+        JSON.stringify({
+          totalRuns: 1, passedRuns: 1, passRate: '100%', meanDuration: 1,
+          fingerprint: computeFingerprint(evalDir, modelConfig as never),
+          contentFingerprint: computeContentFingerprint(evalDir),
+        })
+      );
+
+      // Up to date.
+      expect(runCli(['status'], projectDir).stdout).toContain('up to date');
+
+      // Add a NEW eval (no result) + CHANGE the existing one.
+      const evalDir2 = join(projectDir, 'evals', 'eval-2');
+      mkdirSync(evalDir2, { recursive: true });
+      writeFileSync(join(evalDir2, 'PROMPT.md'), 'do it');
+      writeFileSync(join(evalDir2, 'EVAL.ts'), 'x');
+      writeFileSync(join(evalDir2, 'package.json'), '{"type":"module"}');
+      writeFileSync(join(evalDir, 'EVAL.ts'), 'v2');
+
+      const out = runCli(['status'], projectDir).stdout;
+      expect(out).toContain('new'); // eval-2
+      expect(out).toContain('eval-2');
+      expect(out).toContain('changed'); // eval-1
+      expect(out).toContain('eval-1');
+      expect(out).toContain('to run');
+    });
+  });
 });