Skip to content

Commit 2eef698

Browse files
Version Packages (#45)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent 9f7af62 commit 2eef698

3 files changed

Lines changed: 14 additions & 13 deletions

File tree

.changeset/run-all-and-improvements.md

Lines changed: 0 additions & 12 deletions
This file was deleted.

packages/agent-eval/CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,18 @@
11
# @vercel/agent-eval
22

3+
## 0.3.0
4+
5+
### Minor Changes
6+
7+
- [#44](https://github.com/vercel-labs/agent-eval/pull/44) [`9f7af62`](https://github.com/vercel-labs/agent-eval/commit/9f7af6276ce0f61c79c31ef66cc47b161c0f0028) Thanks [@gaojude](https://github.com/gaojude)! - Add `run-all` command with fingerprinting, failure classification, and housekeeping.
8+
9+
- **run-all command**: Auto-discovers `experiments/*.ts` and runs them all with fingerprint reuse, AI failure classification, auto-retry of infra failures, and housekeeping. Now the default when `agent-eval` is invoked with no arguments.
10+
- **Content fingerprinting**: Computes SHA-256 fingerprints from eval files + config. Skips evals with matching cached results. Safe to extend model arrays or add new evals.
11+
- **Failure classification**: Classifies failed evals as model/infra/timeout using AI via `gateway('anthropic/claude-sonnet-4-5')` with sandboxed tools. Requires `AI_GATEWAY_API_KEY`.
12+
- **Housekeeping**: Removes duplicate results, incomplete results, and empty timestamp directories after each experiment.
13+
- **--smoke flag**: Picks the first eval alphabetically and runs it once per model for quick setup verification.
14+
- **Output naming fix**: Script outputs moved to `outputs/scripts/{name}.txt` to prevent collision with `outputs/eval.txt`.
15+
316
## 0.2.0
417

518
### Minor Changes

packages/agent-eval/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@vercel/agent-eval",
3-
"version": "0.2.0",
3+
"version": "0.3.0",
44
"description": "Framework for testing AI coding agents in isolated sandboxes",
55
"repository": {
66
"type": "git",

0 commit comments

Comments
 (0)