@vercel/agent-eval@1.3.1

Patch Changes

• #170 d6e9c86 Thanks @molebox! - Fix OpenCode explicit model overrides never routing through the AI Gateway. --model anthropic/claude-sonnet-5 was passed to the OpenCode CLI verbatim, which reads the first segment as its provider id — a provider that is not configured in the generated opencode.json (only vercel, the AI Gateway, is) — so every explicit-model run died at session start with "Unexpected server error". The model override is now resolved host-side (vercel/ is prefixed unless the caller already targets vercel/... or a configured extraProviders key) and handed to the runner via input.extra, mirroring codex's openai/ prefixing. Observed models are reported back in the caller's namespace: when the host added the prefix, the runner strips the leading vercel/ from the observation, so observedModel === requestedModel holds for canonical gateway ids and a gateway substitution still surfaces as a clean gateway id. Native-default observations (e.g. vercel/google/gemini-3-pro-preview) and already-prefixed or extra-provider requests are unchanged.

@vercel/agent-eval@1.3.0

Minor Changes

• #165 b35873c Thanks @gaojude! - Incremental eval-staleness workflow, so adopting a changed or new eval doesn't force re-running every experiment.

  • Fingerprint split: each result stores a content-only hash next to the combined (content+config) one. A real eval change is never masked; a benign config change (e.g. a timeout bump, or pinning a judge) is carried forward by refingerprint instead of re-running. Existing fingerprint values are byte-identical, so caches stay valid. (Fixes the previous re-fingerprinting that silently re-stamped every result and hid eval changes.)
  • agent-eval status — read-only: which evals are new vs changed, per experiment (classified by content). --check exits non-zero on any new/changed eval (a simple CI gate); --json emits per-experiment new/changed so a consumer can apply its own "which staleness is acceptable" policy.
  • agent-eval run <experiments...> — run the named experiments' new/changed evals (auto-carries config-only changes first).
  • Bare agent-eval shows status, then (in a terminal) lets you multi-select which experiments to run — it never re-runs everything.
  • Removes run-all and --dry (the run-everything-when-stale behavior). There is no in-framework "acknowledge/keep" — staleness acceptance is the consumer's policy (e.g. filter status --json against an accepted-stale list in CI).

• #164 2905905 Thanks @gaojude! - Pin the agentic LLM judge to a fixed agent + model via ExperimentConfig.judge. By default the expect(environment|transcript) matchers still self-grade with the codegen agent+model; setting judge: { agent?, model } grades every run with one fixed judge — the apples-to-apples choice for cross-model comparisons (judge quality no longer varies with the model under test, and a model never grades itself). When judge.agent names a different agent, its CLI is installed in the sandbox and its key is resolved from its own env var (falling back to VERCEL_OIDC_TOKEN). Pinning is reflected in the eval fingerprint, so pinned runs don't reuse self-graded cached results.

@vercel/agent-eval@1.2.0

Minor Changes

• #162 7021c4b Thanks @gaojude! - Add an agentic LLM-judge matcher for EVAL.ts. Each judge assertion re-invokes the same agent (and model) that did the codegen, in the same sandbox, to evaluate a criterion — then returns pass/fail. No fresh sandbox, no copied evidence.

  import { test, expect } from "vitest";
  import { environment, transcript } from "@vercel/agent-eval/eval";
  
  test("quality", async () => {
    await expect(environment).toSatisfyCriterion(
      "uses Server Components for the product list"
    );
    await expect(transcript).toSatisfyCriterion(
      "diagnosed with DevTools, not trial-and-error"
    );
    await expect(environment).toScoreAtLeast(
      "production-quality error handling",
      0.8
    );
  });

  • Two implicit subjects (environment, transcript) — no paths.
  • You supply only the criterion; the framework owns the prompt + verdict contract.
  • Failures are attributable in the eval output ([judge:environment] FAIL (score): reason).
  • The raw transcript is materialized to a sandbox file so the judge reads it by path (never dumped into a prompt). Framework files under __agent_eval__/ are now gitignored, so they no longer appear in captured generated files.

@vercel/agent-eval@1.1.1

Patch Changes

• #152 c016ea0 Thanks @molebox! - Fix Claude Code prompt being consumed by --allowedTools when webResearch is enabled. The flag is variadic and keeps capturing positionals until the next flag, so even the single comma-separated token from 1.1.0 swallowed the trailing prompt ("Input must be provided either through stdin or as a prompt argument when using --print", verified live on claude 2.1.112). --allowedTools is now emitted before the always-present --dangerously-skip-permissions, which terminates the variadic capture before the prompt. Default-off argument construction is unchanged.

@vercel/agent-eval@1.1.0

Minor Changes

• #150 084d895 Thanks @molebox! - Add an opt-in webResearch option that enables each agent's web research tools so recommendation evals can produce citation/source data. Default is off: command construction is byte-identical to previous releases for existing consumers.

  The option is available on AgentRunOptions and on ExperimentConfig, and is forwarded by runExperiment/runSingleEval, so both direct executeAgent callers and experiment-config consumers can use it.

  When enabled:

  • Claude Code: allows WebSearch and WebFetch via a single comma-separated --allowedTools value (the flag is variadic — the space-separated form in #141 consumed the trailing positional prompt as a tool name, which is what broke all CLI evals and forced the #144 revert).
  • Codex: sets tools.web_search = true in the generated profile config.
  • OpenCode: sets OPENCODE_ENABLE_EXA=1 and allows the websearch/webfetch tools.

Note: published to npm by the Release workflow, but the run failed post-publish with npm error code E401 (invalid NPM_TOKEN), so this tag/release was created manually.

@vercel/agent-eval@1.0.1

Patch Changes

• #148 b4841d6 Thanks @molebox! - Fix OpenCode observed model extraction for OpenCode >= 1.17.0. The log-scrape source (service=llm ... providerID= modelID= lines) was removed in OpenCode 1.17.0's logging rewrite, which caused native-default runs to report no observed model. The adapter now falls back to opencode export <sessionID>, reading providerID/modelID from the exported assistant message. The legacy log scrape is kept as the first, cheaper source for older CLI versions.

@vercel/agent-eval@1.0.0

Major Changes

• #146 aa66c4d Thanks @molebox! - Change omitted model config to use the underlying agent CLI's native default instead of agent-eval's hardcoded adapter defaults, and record observed runtime model metadata when available.

@vercel/agent-eval@0.14.5

Patch Changes

• #144 450bed3 Thanks @allenzhou101! - Revert "Enable source-capable web tools for agent runs" (#141), which broke CLI evals.

@vercel/agent-eval@0.14.4

Patch Changes

• #141 2d27942 Thanks @molebox! - Enable source-capable web tools for recommendation eval agent runs.

@vercel/agent-eval@0.14.3

Patch Changes

• #140 a9efa3a Thanks @allenzhou101! - Default the Codex profile's model_reasoning_effort to "medium".

  The Codex CLI's own default is "low", which gpt-5.2-codex (the default
  Codex model) rejects with Unsupported value: 'low' is not supported with the 'gpt-5.2-codex' model. Supported values are: 'medium'.. The Codex
  adapter now writes model_reasoning_effort into the generated profile so
  fresh codex exec runs against the AI Gateway succeed out of the box.
  Callers can still override per-run via
  model: "gpt-5.2-codex?reasoningEffort=high".