From ad9ccbc3a676e7d5f11bbf4b4ade82b883edab24 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 09:14:37 +0000 Subject: [PATCH 1/9] Add online-mind2web Harbor adapter (Kernel env + cua agent, WebJudge) Generates Harbor task dirs from the gated osunlp/Online-Mind2Web dataset (300 live web tasks) and grades them with WebJudge. Builds on the shared cua_harbor agent + cua-bench-task entrypoint; adds only the dataset->task generator and the verifier. - adapter.py maps each row to a task dir: kernel.json (start_url normalized to https://, stealth, 1280x1024 viewport), instruction.md, task.toml, tests/task.json. Mirrors the cua loader's field mapping + skip-malformed. - WebJudge verifier is a self-contained Node bundle (recovered webjudge/ prompts placed verbatim; Anthropic judge over fetch) that runs in-VM, reconstructs the trajectory from the shared core's run.jsonl + shots/ + answer.txt, and writes the reward. - Mocked unit tests (pytest, vitest), ruff + tsc clean. Generated tasks, build artifacts, and caches are gitignored. Co-Authored-By: Claude Opus 4.7 --- .../adapters/online-mind2web/.gitignore | 17 + benchmarks/adapters/online-mind2web/README.md | 57 + benchmarks/adapters/online-mind2web/SMOKE.md | 67 + .../online-mind2web/judge/package-lock.json | 2466 +++++++++++++++++ .../online-mind2web/judge/package.json | 25 + .../online-mind2web/judge/src/artifacts.ts | 102 + .../online-mind2web/judge/src/judge.ts | 97 + .../online-mind2web/judge/src/model.ts | 75 + .../online-mind2web/judge/src/prompts.ts | 139 + .../online-mind2web/judge/src/types.ts | 46 + .../online-mind2web/judge/src/webjudge.ts | 66 + .../judge/test/artifacts.test.ts | 132 + .../judge/test/webjudge.test.ts | 119 + .../online-mind2web/judge/tsconfig.json | 17 + .../online-mind2web/judge/tsdown.config.ts | 15 + .../online-mind2web/judge/vitest.config.ts | 12 + .../adapters/online-mind2web/pyproject.toml | 28 + .../src/online_mind2web/__init__.py | 5 + .../src/online_mind2web/adapter.py | 245 ++ .../src/online_mind2web/main.py | 81 + .../task-template/instruction.md | 10 + .../task-template/instruction.nourl.md | 10 + .../task-template/solution/solve.sh | 8 + .../online_mind2web/task-template/task.toml | 28 + .../task-template/tests/test.sh | 23 + .../online-mind2web/tests/test_adapter.py | 145 + 26 files changed, 4035 insertions(+) create mode 100644 benchmarks/adapters/online-mind2web/.gitignore create mode 100644 benchmarks/adapters/online-mind2web/README.md create mode 100644 benchmarks/adapters/online-mind2web/SMOKE.md create mode 100644 benchmarks/adapters/online-mind2web/judge/package-lock.json create mode 100644 benchmarks/adapters/online-mind2web/judge/package.json create mode 100644 benchmarks/adapters/online-mind2web/judge/src/artifacts.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/src/judge.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/src/model.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/src/prompts.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/src/types.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/src/webjudge.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/test/webjudge.test.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/tsconfig.json create mode 100644 benchmarks/adapters/online-mind2web/judge/tsdown.config.ts create mode 100644 benchmarks/adapters/online-mind2web/judge/vitest.config.ts create mode 100644 benchmarks/adapters/online-mind2web/pyproject.toml create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/__init__.py create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/main.py create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/solution/solve.sh create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml create mode 100644 benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh create mode 100644 benchmarks/adapters/online-mind2web/tests/test_adapter.py diff --git a/benchmarks/adapters/online-mind2web/.gitignore b/benchmarks/adapters/online-mind2web/.gitignore new file mode 100644 index 0000000..98d3e01 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/.gitignore @@ -0,0 +1,17 @@ +# Generated task dirs + caches (never committed) +.tasks/ +jobs/ +trials/ +dataset/ +*.cache/ + +# Python +__pycache__/ +*.egg-info/ +.venv/ +.pytest_cache/ +.ruff_cache/ + +# Node judge bin build artifacts +judge/dist/ +judge/node_modules/ diff --git a/benchmarks/adapters/online-mind2web/README.md b/benchmarks/adapters/online-mind2web/README.md new file mode 100644 index 0000000..072e3e9 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/README.md @@ -0,0 +1,57 @@ +# online-mind2web — Harbor adapter + +Converts [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web) +(300 live web tasks, OSU NLP Group) into Harbor tasks that run on the Kernel +environment with the cua agent and are graded by WebJudge. + +Each upstream row is a `(instruction, start_url)` pair with no gold answer; +grading is trajectory-based. The adapter builds on the shared core under +`benchmarks/` (the `cua_harbor` agent + the `cua-bench-task` Node entrypoint) +and adds only the dataset→task generator and the WebJudge verifier. + +## Layout + +``` +adapters/online-mind2web/ + pyproject.toml uv project "harbor-online-mind2web-adapter" + src/online_mind2web/ + adapter.py OnlineMind2WebAdapter: rows -> task dirs + main.py CLI: --output-dir --limit --overwrite --task-ids + task-template/ task.toml, instruction.md(+nourl), kernel.json, solve.sh, tests/test.sh + judge/ self-contained WebJudge Node bin (bundled, runs in-VM) + src/ webjudge.ts + prompts.ts (recovered, tested) + model.ts (Anthropic) + judge.ts (CLI) + tests/test_adapter.py mocked adapter tests (no network) +``` + +## Generate tasks + +The dataset is gated on HuggingFace. A pre-fetched copy at `/tmp/om2w-real.json` +is used when present; otherwise set `HF_TOKEN` (after accepting the dataset +terms) and the adapter fetches + caches it. Build the judge bundle first — the +adapter copies it into each task's `tests/`. + +```bash +cd benchmarks/adapters/online-mind2web/judge && npm install && npm run build +cd .. && PYTHONPATH=src python -m online_mind2web.main --limit 20 --overwrite +``` + +Generated tasks land in `.tasks/` (gitignored). + +## Grading + +`tests/test.sh` runs the bundled `judge.js` inside the Kernel VM (which ships +`node` + global `fetch`), reading the agent's artifacts under `/logs/agent` +(`answer.txt`, `run.jsonl`, spilled `shots/*.png`), reconstructing the WebJudge +trajectory, grading it against an Anthropic judge model, and writing a single +reward float to `/logs/verifier/reward.txt`. The judge model and score +threshold are set in `[verifier.env]` (`JUDGE_MODEL`, `SCORE_THRESHOLD`). + +## Run on Harbor + +```bash +cd benchmarks && uv sync && (cd node && npm install && npm run build) +uv run harbor run -p adapters/online-mind2web/.tasks -e kernel \ + --environment-kwarg pool_size=8 \ + --agent-import-path cua_harbor:CuaHarborAgent \ + -m anthropic/claude-sonnet-4-6 --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY -n 6 +``` diff --git a/benchmarks/adapters/online-mind2web/SMOKE.md b/benchmarks/adapters/online-mind2web/SMOKE.md new file mode 100644 index 0000000..8717177 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/SMOKE.md @@ -0,0 +1,67 @@ +# online-mind2web — smoke notes + +The live 20-task Harbor smoke was **not run** in this session (scope: implement ++ unit-test the adapter only; the parent runs the live smoke after review). +What was validated, and the learnings that shaped the implementation, are below. + +## Validated without the live harbor run + +- **Judge against the live Anthropic API.** The bundled `judge.js` ran + standalone against `anthropic:claude-sonnet-4-6`: it made the WebJudge + KEY_POINTS + FINAL model calls, parsed the verdict, and wrote + `reward.txt` + `grading_details.json`. The `model.ts` fetch client, the + recovered three-stage pipeline, the parsers, and the reward writer all work + end-to-end against the real provider. +- **Kernel VM runtime probe.** A throwaway browser session confirmed the VM + ships `node v22.23.1` (with global `fetch`), `python3 3.10`, and `curl`, and + that `/logs/agent` + `/logs/verifier` are writable and ESM bundles run. This + is the fact the verifier design hinges on (see learnings). +- **Generation end-to-end.** `python -m online_mind2web.main --limit 3` against + the cached dataset produced full task dirs (`instruction.md`, `task.toml`, + `environment/kernel.json`, `tests/{test.sh,judge.js,task.json}`, + `solution/solve.sh`) with correct `start_url` normalization and slugged ids. +- **Unit tests / lint / typecheck.** `ruff check` clean; 9 mocked Python adapter + tests pass (no network — dataset injected); judge `tsc --noEmit` clean; 13 + vitest tests pass (parsers, `gradeWithWebJudge` with a scripted judge, and + `loadTrajectory`/`loadTask` against a real on-disk `/logs/agent` layout). + +## Learnings that changed the design vs `map-online-mind2web.md` + +The design doc predates the finalized shared core (PR #40) and assumed two +artifacts that do not exist; the implementation reconciles to what the shared +core actually emits. + +1. **No `trajectory.bench.json`, no `@onkernel/cua-bench` `grade.js`.** The + shared `node/src/sink.ts` writes `answer.txt`, `run.jsonl`, and spilled + `shots/shot-.` (1-indexed) — not a purpose-built grader index, and + the entrypoint package is `cua-bench-task`, not `@onkernel/cua-bench`. The + verifier therefore reconstructs the WebJudge `Trajectory` directly from + `run.jsonl` (pairing each `tool_result`'s spilled shot with the originating + `assistant` tool call as the "action") + `answer.txt` (`judge/src/artifacts.ts`). +2. **The Kernel VM does have `node`.** The benchmarks README says the base VM + "ships no Node", which is true for the cua *agent* (its npm package isn't + installed in-VM, so the agent entrypoint runs on the host). But a raw `node` + binary is present, so the verifier's `test.sh` runs the WebJudge in-VM. The + judge is bundled to a single self-contained ESM file (tsdown, no externals, + ~16 kB) and copied into each task's `tests/` so it travels with the uploaded + tests dir and needs no install at verify time. +3. **Anthropic judge, not OpenAI.** No `OPENAI_API_KEY` this session, so the + WebJudge backbone is `anthropic:claude-sonnet-4-6` via a minimal Anthropic + Messages client over global `fetch` (`judge/src/model.ts`) instead of the + recovered cua-ai `piJudgeModel`. `JUDGE_MODEL` / `SCORE_THRESHOLD` stay + configurable in `[verifier.env]`. The recovered `webjudge.ts` / `prompts.ts` + are otherwise placed verbatim (they are line-checked against upstream and + carry the parser-coupled literal markers). +4. **Answer-file path = `/logs/agent/answer.txt`.** Matches the shared-core + contract (`cua_harbor.constants.ANSWER_FILE`) and the example task's + `test.sh`, resolving the open item in the design doc. + +## What the live smoke should still surface (to capture next round) + +- Pass rate / per-task reward across ~20 tasks with `pool_size=8` (fall back to + `pool_size=1` if pools 403 on this account; note it, don't fail the smoke). +- Failure taxonomy: env-vs-task, live-site drift / anti-bot walls (stealth is on + in every `kernel.json`), judge disagreement, adapter bugs. +- Whether `claude-sonnet-4-6` as the judge backbone tracks the published + WebJudge(o4-mini) success rate within a tolerance band (live-site drift makes + exact parity impossible; pin judge + viewport + threshold and record the date). diff --git a/benchmarks/adapters/online-mind2web/judge/package-lock.json b/benchmarks/adapters/online-mind2web/judge/package-lock.json new file mode 100644 index 0000000..c5679c3 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/package-lock.json @@ -0,0 +1,2466 @@ +{ + "name": "online-mind2web-webjudge", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "online-mind2web-webjudge", + "version": "0.1.0", + "license": "MIT", + "bin": { + "online-mind2web-webjudge": "dist/judge.js" + }, + "devDependencies": { + "@types/node": "^22.10.0", + "tsdown": "^0.22.2", + "typescript": "5.9.3", + "vitest": "^3.2.4" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@babel/generator": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-8.0.0.tgz", + "integrity": "sha512-NT9NrVwJsbSV6Y2FSstWa71EETOnzrjkL5/wX3D2mYHtKM+qvqB1DvR4D0Setb/gDBsHzRICifwEWMO8CnTF6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^8.0.0", + "@babel/types": "^8.0.0", + "@jridgewell/gen-mapping": "^0.3.12", + "@jridgewell/trace-mapping": "^0.3.28", + "@types/jsesc": "^2.5.0", + "jsesc": "^3.0.2" + }, + "engines": { + "node": "^22.18.0 || >=24.11.0" + } + }, + "node_modules/@babel/helper-string-parser": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-8.0.0.tgz", + "integrity": "sha512-6mJgmFFFIIO82vvoLt9XtRC7/TkzXfts1t/SpRX4IHSzMgqoPYCWesVu1udUPUWioAE/2fcG6WuI8zrkE1gwrg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^22.18.0 || >=24.11.0" + } + }, + "node_modules/@babel/helper-validator-identifier": { + "version": "8.0.2", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-8.0.2.tgz", + "integrity": "sha512-9Fr9QeyCAyi1BR1jKZ6uYQ24EIhQUx5ReHfQU7drOE+TPOb+w11/dsqLkMOT2U29OdCT71XajrOT8xDc1C7orA==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^22.18.0 || >=24.11.0" + } + }, + "node_modules/@babel/parser": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-8.0.0.tgz", + "integrity": "sha512-aLxAE+imI9bCcyaPrUDjBv3uSkWieifjLe0kuFOZF0zli0L6GCsTmsePnTr55adbIAgYz2zhN1vnFimCBUYcRQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/types": "^8.0.0" + }, + "bin": { + "parser": "bin/babel-parser.js" + }, + "engines": { + "node": "^22.18.0 || >=24.11.0" + } + }, + "node_modules/@babel/types": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/@babel/types/-/types-8.0.0.tgz", + "integrity": "sha512-K8ponJDxBwDHigkeFqaqT5wLGl4bTlwMafR8k7b5CPxr6Ww+UG9ls8Yx6Tcpboxu97eeGVEEyKcHmEyOwN1vSw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/helper-string-parser": "^8.0.0", + "@babel/helper-validator-identifier": "^8.0.0" + }, + "engines": { + "node": "^22.18.0 || >=24.11.0" + } + }, + "node_modules/@emnapi/core": { + "version": "1.11.1", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", + "integrity": "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.2", + "tslib": "^2.4.0" + } + }, + "node_modules/@emnapi/runtime": { + "version": "1.11.1", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.1.tgz", + "integrity": "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@emnapi/wasi-threads": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.2.tgz", + "integrity": "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.1.tgz", + "integrity": "sha512-Svl7tq8k/08+p6CXPpRjQ1fKX+1odH/BQbb48fV6fj3CWHhsoIOoY87w1oHXm0qEpkIK3ZfVgp0hed3XBXzXMQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.1.tgz", + "integrity": "sha512-0k2F129Xdio1TdJfzJ8sy1Q47vUD2NnwdhiAf7drUN1EBTfPf4hsFCtmMgu/6m8JSzsBrlmVjudMBQqOfG8usQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.1.tgz", + "integrity": "sha512-34EGEbCIAgosYz6goLcopX6Mo7NyGv9tfwEM2/7Ce2VcVRk568iSvniGWcUXIy7wEDR1wzolcxcriFVrWYcwBg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.1.tgz", + "integrity": "sha512-dbwY7ltSMDWsRatcRpCnES4F+im88OCUgGZjy52shC7GqHRE/cYlxNbB4Z4UpJswpcc4Qxd2oE/ufM0p61IKng==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.1.tgz", + "integrity": "sha512-TZbWkQY7kvTAXbXUT7uVACR5cMHsDiSz9z7ZKAX/RTq/WJEk3QyRr0wZpNhBDX+/0CtdqUIJlOiodQcta6tY3Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.1.tgz", + "integrity": "sha512-zfdzgK9ACBNZLI/CyHTOx81SyNbM6YXn7rxSgX97VjyiPl9W1i4Ka4fgKECEoFCKGpvBj5qArWIGgQjOwkgskQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.1.tgz", + "integrity": "sha512-wG2EA8ENdEI0qhkSZMjfqrdY+ziCYCPMmtZjjIwOmXFjmyzEHn+UUxk5of+SYsjtfs3VpnlC7QLzSI5hY/rOAw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.1.tgz", + "integrity": "sha512-i7dZ9vQgnvSCzi/rYCXNgtF/U+eKZNJBzu3eTQbRgHnM7tNSizLOkRFAl3qzVc/Op/u5YkHHa4pf/3DOYHthLQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.1.tgz", + "integrity": "sha512-qVXBOHQS+d5Y722GwJzJUtOLlX7km3CraOaGormF1pDtPd2C/l1SHRPgjLunLGe51Sh5YYWKMFDyV4SxgMQYTQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.1.tgz", + "integrity": "sha512-yHs+0uc8+nvEAfAfxrWQKK5peSNzBc4PegcMO0EJ2hT71uA7vB8Ihg2e77R2P7SG5uYjPbHlLLmve4LLLRCf0g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.1.tgz", + "integrity": "sha512-d1z4ZuP0ajrfz/FhGT4vv278rX8KnPPJx8i5+AtK7TYbx9Le9F1hyzurZpkEyjkGa9dUGhQow4C1NmeGvqxN2w==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.1.tgz", + "integrity": "sha512-M5sRjUVZrkm1OAPR3dlOYzNmN+loZKGVi1VUQGrwuqLcbR6qeAz+famMhjASeH3YVKvZz+zT1jlh/keC3Rj/lg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.1.tgz", + "integrity": "sha512-mRObBZeHh2OxcBFPWE/FjylkRgZdYuiTR3vaTozquCGOH14iP9oN4x4Ge81CoIDYQrXmIxpFumJBu5MtZpnQJQ==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.1.tgz", + "integrity": "sha512-slScBsMAb3GFDcdrCgLwZtPYRoH2H/youv10QiZyRjmsP48fznoveWytSgCI/R0ZcUgpc0ZhIUEx6LHts8yrfQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.1.tgz", + "integrity": "sha512-kw0owk1o0GFETUJyW0jc0G4Yzs0BHZn0JDZ8JRT088vjJYX777BAs1fDGxAC+q831qOs2DTC96mNsG2opdfyyQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.1.tgz", + "integrity": "sha512-/lAIjX8aYFRByhh6L5rYtPEDRqa9de/4V/juOXcta5frjvzXO4/sqEtyytse0g3zZFuWu5cDN0MkLz2qRDD2Ag==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.1.tgz", + "integrity": "sha512-u/anNYF2mmVOEDwLtnQ1wOr3EZ9sTNGLWrsYGYwHWzGA3Si84IOkHXlbWTD1NB+9/1lcnweYKO54uhxZydNzfA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.1.tgz", + "integrity": "sha512-oks0DYbLwWMmaakTsCb+zL4E+aHRVLom9IJZOAthMQEPiQmydXHkziYEsGYRx0uNV/IjEKGAV941JzH02pflqw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.1.tgz", + "integrity": "sha512-aeL6lAnN89Hz43Mlh1G8ARasbuoYvSITDEx0tHh5b7jJnHcssqgjy9Yx430GDpmCa6OyrKoS0aNRjKundRizGg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.1.tgz", + "integrity": "sha512-MEFJe5C3R8pwXdZ5Y21oo6m7ePiS0d9pWucn99O/wvyJZChoIQKrQDxKrGeW8F5+T0okTHesAmDeiHDTIq0V/Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.1.tgz", + "integrity": "sha512-i/ZLIOafE0Z8cI/XANJAixoJL/uRAoS2xOA3rb0xN+KK0K177cMAsQYkzHtBrtMXAKuAc7HGgcWiZ/sRC1Nxgw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openharmony-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.1.tgz", + "integrity": "sha512-ge+Z7EXFNt2BO1oAMsVpiQ8EwndV9i1xXerAeTIK7AtPs3bKFXQM7nlRxDSIUIMeueR1CNXxqztLzdNeReKBJg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.1.tgz", + "integrity": "sha512-BEjgtECkL3vY+SaSQ6nzVfiALUeFxpawyp8Jmf5PtYhf1Ug40N1h/hxlhts+f1FvSvarEigdxS3BlSMI2PJLcQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.1.tgz", + "integrity": "sha512-lCv9eK/H6ZJWbE7bh2nw54CZ9M2nupBxJcTsdk/QQnWkdSjKGuxmmH8/GWrlT1eMmZfn4dGcCjRte397WqfQXA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.1.tgz", + "integrity": "sha512-zvb/mB2bSCoJOpoCBgYKKpX6YM6mJBlBUVUtVj41DlZJVEB6/0CKlRYxP5wWl1C1ILiCoAU5wZZ4q1P3qeS6Eg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.1.tgz", + "integrity": "sha512-bm4Mowrv+GXMlpWX++EcXw/iLyd1o3+bJkC2DkWXYVvgZCqD/bSj9ctZeAMC3cIxgjRVR2Dufaiu4YPxr5gW1A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@jridgewell/gen-mapping": { + "version": "0.3.13", + "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", + "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.0", + "@jridgewell/trace-mapping": "^0.3.24" + } + }, + "node_modules/@jridgewell/resolve-uri": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", + "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.5.5", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", + "integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==", + "dev": true, + "license": "MIT" + }, + "node_modules/@jridgewell/trace-mapping": { + "version": "0.3.31", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", + "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/resolve-uri": "^3.1.0", + "@jridgewell/sourcemap-codec": "^1.4.14" + } + }, + "node_modules/@napi-rs/wasm-runtime": { + "version": "1.1.6", + "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.6.tgz", + "integrity": "sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@tybys/wasm-util": "^0.10.3" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "peerDependencies": { + "@emnapi/core": "^1.7.1", + "@emnapi/runtime": "^1.7.1" + } + }, + "node_modules/@oxc-project/types": { + "version": "0.137.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.137.0.tgz", + "integrity": "sha512-WT+Gb24i8hmvo85AIv2oEYouEXkRlKAlT9WaCa3TfLgNCN+GhrJOGZuIlMouAh38Qe4QOx26eUOVsq70qXrywA==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/Boshen" + } + }, + "node_modules/@quansync/fs": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@quansync/fs/-/fs-1.0.0.tgz", + "integrity": "sha512-4TJ3DFtlf1L5LDMaM6CanJ/0lckGNtJcMjQ1NAV6zDmA0tEHKZtxNKin8EgPaVX1YzljbxckyT2tJrpQKAtngQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "quansync": "^1.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + } + }, + "node_modules/@rolldown/binding-android-arm64": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.1.3.tgz", + "integrity": "sha512-DT6Z3PhvioeHMvxo+xHc3KtqggrI7CCTXCmC2h/5zUlp5jVitv7XEy+9q5/7v8IolhlioawpMo8Kg0EEBy7J0g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-arm64": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.1.3.tgz", + "integrity": "sha512-0NwgwsjM7LrsuVnXMK3koTpagBNOhloc/BNjKqZjv4V5zI5r13qx69uVhRx+o5Z0yy4Hzq+lpy7TAgUG/ocvrw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-x64": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.1.3.tgz", + "integrity": "sha512-YtiBp4disu6V560loT6PjMdiRaWmVvDNrUunAalbiFx2ggeJwxdAsgZMcoGP17uyAsTwAj5V1niksxlHnVQ1Sw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-freebsd-x64": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.1.3.tgz", + "integrity": "sha512-yD3EkEdXk2LypPxnf/kSZHirarsI8gcPzc62SukhR9VJTyvV+F9Q/GxWNuCojc7sXyuVC4DxRGhdDK4X8VSsbw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm-gnueabihf": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.1.3.tgz", + "integrity": "sha512-c+8vieQbsD7HNAHKIA34w0GJ9FedFFuJGD+7E6vz7Q3uqAIugL5p45fhlsj4UaAsHpcmlqugBWMhA0/j7o0sIg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-gnu": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.1.3.tgz", + "integrity": "sha512-50jD0uUwLvur7Zz9LHz17kaAdTPjn5wN93hEgjvmYFRZwiR7ZJYovTd5ipyWJDAnXKvZ+wgc+/Ika6dwSF5OcA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-musl": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.1.3.tgz", + "integrity": "sha512-BO9+oPL8K9poZJBfYPsXNtYjPE5uM3qeehT3aFcW4LITOl+iSqhp0abzjR2nWBUNjIZeKXjAEWBZ64WjNoHd6w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-ppc64-gnu": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.1.3.tgz", + "integrity": "sha512-f3VpLB1vQ0Eo6ecr/6cekLnvYMFF4YBFoVGkfkvPLq1bAkbAwHYQPZKoAmG6OJyTcxxoC+AvezGx/S1obNC0Mw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-s390x-gnu": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.1.3.tgz", + "integrity": "sha512-AmurZ26Pqx/RI9N1gzEOCklkKXl927yjfXWUUS0O7Puh8ARM/Ob8qfrD3qnWksScdw6cSrW5PSHE9DyLu7+PtA==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-gnu": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.1.3.tgz", + "integrity": "sha512-JJpqs8bRGITDOdbkNKnlojzBabbOHrqjSvDr0IVsZObE1lBcPjxItUEY9eWIDbxaJ3cGrXPWGfGkIxFijg/URg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-musl": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.1.3.tgz", + "integrity": "sha512-rSJcdjPxzA/by/6/rYs+v+bXU7UjvnbUWz8MJb6kh6+knqB1dCrtHg0uu7C/4haqJvqdkYHQ5IGn+tCH9GLW/g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-openharmony-arm64": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.1.3.tgz", + "integrity": "sha512-hQ3/PYkDJICgevvyNcVrihVeqq7k1Pp3VZ9lY+dauAYUJKO+auqApvANhvR1An9BhmqYKvW2Mu1F9u4DXSMLxQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-wasm32-wasi": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.1.3.tgz", + "integrity": "sha512-Elcv/BtML9lXrV6JuKITc/grN2kYV9gjsQpW8Jfw4ioK0TOkjBjye0nnyqQNy9STNaI20lXNaQBRrD5gSgR0Yg==", + "cpu": [ + "wasm32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/core": "1.11.1", + "@emnapi/runtime": "1.11.1", + "@napi-rs/wasm-runtime": "^1.1.6" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-arm64-msvc": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.1.3.tgz", + "integrity": "sha512-2DrEfhluH9yhiaFApmsjsjwrSYbNcY1oFTzYSP1a535jDbV98zCFanA/96TBUd0iDFcxGmw9QRExwGCXz3U+/g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-x64-msvc": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.1.3.tgz", + "integrity": "sha512-OL4OMk7UPXOeVGGd3qo5zJyPIljf4AFgk5QAkPPS+OoLuOOozhuaQGC18MxVTnw/06q93gShAJzlwnSCY9YtqA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz", + "integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.62.2.tgz", + "integrity": "sha512-6o7ZLZK+BeenkZCFNDXqpbjw9bD6nuWonvS/lwQJp7NoVVxm6p3qE7qQ5jGuBjiFsgvqjD8mZAU5oWxTmbOeOg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.62.2.tgz", + "integrity": "sha512-BaH7BllCACHoH1LguOU56UItGfUWjujlO65kS9LAodViaN4bwIKd7oeW/ZHJ/4ljr/7MIiENnNy3HJ0zXv8Zkw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.62.2.tgz", + "integrity": "sha512-v39RCCvj4He82I9sFmk+M1VZ0PLM9sfsLVikjfx2hYBNALhrrOR2D3JjQA6AhlaSOgcR+RzrKY7e1+bT6SUO/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.62.2.tgz", + "integrity": "sha512-yl0y2vq3S3lHeuXhEdss6TWfKW8vkujImO12tn4ZkG/4oghr09LvdYm2RElVjokTQiUvDUGXLGsYeLqUMCKpGA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.62.2.tgz", + "integrity": "sha512-tT4pvt4qXD+vEoezupCWi+a1F0vvDiksiHc+PxRlYTOH1I6/X4id9jPxTP+Fg+545euaFT1jJVs4CEdHZAU1vw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.62.2.tgz", + "integrity": "sha512-6nU5F2wCW+qvCBhTn1pdIU3bzsIoF7EUwsCDRxilWGprQR6yd508YnH9+OKFCwpfS8pjZqDUmnCAr7exax0XCg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.62.2.tgz", + "integrity": "sha512-n1GJHPOvpIfhi3TmrCeh6S6URt9BFCt0KQE3qvexyGCTAKpR4Lg+eWvNZEqu7epxwus/8ElT3hacYEucm49SZg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.62.2.tgz", + "integrity": "sha512-JqgflS8wEB+UXV/vS1RpRbifGBeN4D5lz8D8oOFbFZw4vedvdOgCFAjfBmIMdW3yL10XpQQ0Ambepw6MXrhOnA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.62.2.tgz", + "integrity": "sha512-wnFJkogWvN4jm/hQRF2UBaeUmk20j5+DmHvoyWii2b8HJDyvz1MF2OU/6ynXt2KR63rbZLWkFpoytpdc/yBuSA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.62.2.tgz", + "integrity": "sha512-HVu2bp0zhvJ8xHEV9+UUs7S90VadmBSY3LcIMvozbPo4AuMGDWlz3ymHLHZPX4hR67TKTt8Qp5PJ5RBg/i+RMQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.62.2.tgz", + "integrity": "sha512-mQqqAV8QaoSgr9I2fKDLY2BAVvmKjWoGiu/cSYQonsLvtqwEn1E4QYfnCOcp5zoEqNhsDYin1s6jx/VJmrxlZg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.62.2.tgz", + "integrity": "sha512-IxKLoxCQ2IWi6bT2akyDUBGsOImDKB+sPp4EsTmwFQ/fMwpCKm8uLSSgP/Kx/QYUgKis6SEZ5/Nlhup0DIA0PQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.62.2.tgz", + "integrity": "sha512-Mk5ha2RQSgyFfmYYLkBpPnUk8D8FriBxesO1u9O75X0mHgXL1UQcH5Itl2lurWL2tj0RxV9b9tJgipac0hRY9A==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.62.2.tgz", + "integrity": "sha512-CjvEnqJL/0/TQ3TXX3OPIJ/kmBellrWd4heXUmHeJlTnmwjKpSJzoehLaL6Xk0ZnMHBu9dZuFADNOrtjF4v+2w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.62.2.tgz", + "integrity": "sha512-1SiZbzwdkaDURsew/tSOrooKiYy7EQGT6m8ufavAi9NEyQb/6VuIxFXAL1fqa4iZe3g4NbNk4P7J32z2tw5Mgg==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.62.2.tgz", + "integrity": "sha512-nQts12zJ3NQRoE6uYljOH89v7szzLDvG2JD/vsX+vGXU8w/At1GowTZ5/7qeFQ8m7L55rpR8Okugnuo5bgjy2Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.62.2.tgz", + "integrity": "sha512-E9/ll019jhPIJgpzfZoIkBGhcz+kKNgVWYRY0zr9srBdPPFVpvOKW8VaJKUbeK+eZXyQF9ltME+Kk6affeaPgg==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.62.2.tgz", + "integrity": "sha512-5BqxR/pshjey51iliyzTD5Xi3EN0aLmQ2lZ3lvefVV9c82BvrLo2/6OT55iifpWBufs6kdwWbuOKS841DrmK9A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.62.2.tgz", + "integrity": "sha512-uNN83XxQrRAh/w0/pmAfibcwyb6YWt4gP+dpnQKPVJshAloQ785ii8CT8ZCIxkGg9opVsvAlGhFitSm6D1Jjpg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openbsd-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.62.2.tgz", + "integrity": "sha512-srjEIxSH3LRnJN6THczDHWQplqEMFiAJrTab0msUryh9kwNpkICf3Ea6q6MN/2cZwRFUNx5w+h6Hpi4QuHS6Zg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.62.2.tgz", + "integrity": "sha512-8hOJnxgbyObnCm5AlRA3A931xX19xq80RjVTKgJOvEKWqJruP/Uf12IbAOaDjjEXYRewwHLfmF0YRIdK3OwKWA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.62.2.tgz", + "integrity": "sha512-mmF4AY1i0hG/bLWUctUq59gtmgaSIRa3cu/A3JFRp/sCNEme2bgDEiDS22P9FbnJB8NJNF4jPJiSP5RHQpUTDg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.62.2.tgz", + "integrity": "sha512-DZgkknc6jhHrk46V25vbAM0zZkyP0nSDkJB8/dRkLTxv470dOmWDqGoEJl/9A0dFfS7yE3REOwNDxpHwSLSt0Q==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.62.2.tgz", + "integrity": "sha512-T6xr6ucWSFto+VGajA8YH26LdpHRuP4YLHEKAtCWvJDOlnmWcDZVCI2Jmjr+IFHDlt2zRaTAKE4tfjTaWLgJBg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.62.2.tgz", + "integrity": "sha512-BfzEnDJOt9T8M989/lA37EcJgat01wLRnoi5dQf3QzOH7jzpqTAzdDbVfRljVr5r+jzKqpbHeyOfAaXxAd0PAA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@tybys/wasm-util": { + "version": "0.10.3", + "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.3.tgz", + "integrity": "sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@types/chai": { + "version": "5.2.3", + "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", + "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/deep-eql": "*", + "assertion-error": "^2.0.1" + } + }, + "node_modules/@types/deep-eql": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz", + "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/estree": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz", + "integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/jsesc": { + "version": "2.5.1", + "resolved": "https://registry.npmjs.org/@types/jsesc/-/jsesc-2.5.1.tgz", + "integrity": "sha512-9VN+6yxLOPLOav+7PwjZbxiID2bVaeq0ED4qSQmdQTdjnXJSaCVKTR58t15oqH1H5t8Ng2ZX1SabJVoN9Q34bw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "22.20.0", + "resolved": "https://registry.npmjs.org/@types/node/-/node-22.20.0.tgz", + "integrity": "sha512-QWlFW2wf3nTjC13/DqRnBpR4ZO36VJH/JVBkA/vcnmbTBNQIlnObqyqZE1tUR7+Ni23Lda8R1BxMfbXRpCUx5g==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~6.21.0" + } + }, + "node_modules/@vitest/expect": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.6.tgz", + "integrity": "sha512-1+7q9BtaKzEmO+fmNT3kYvoNn5Y71XWAx2Q5HRim4tTVRQVRv4uJFAQ5FbK0OPUeNP/WmVCpxYxoJdvuHVjzBQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/chai": "^5.2.2", + "@vitest/spy": "3.2.6", + "@vitest/utils": "3.2.6", + "chai": "^5.2.0", + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/mocker": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.6.tgz", + "integrity": "sha512-EZOrpDbkKotFAP7wPAQV1UIyoGOk4oX7ynWhBhLB7v+meMHbQhU16oPpIYGTTe4oFlhpryGpgpcZP/sin3hYuw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/spy": "3.2.6", + "estree-walker": "^3.0.3", + "magic-string": "^0.30.17" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "msw": "^2.4.9", + "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" + }, + "peerDependenciesMeta": { + "msw": { + "optional": true + }, + "vite": { + "optional": true + } + } + }, + "node_modules/@vitest/pretty-format": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.6.tgz", + "integrity": "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/runner": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.6.tgz", + "integrity": "sha512-HYcoSj1w5tcgUnzoF0HcyaAQjpA1gj9ftUJ7iSJSuipc02jW9gKkigwZbjFldAfYHA1fa8UZVRftdMY5msWM9Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/utils": "3.2.6", + "pathe": "^2.0.3", + "strip-literal": "^3.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/snapshot": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.6.tgz", + "integrity": "sha512-H+ZjNTWGpObenh0YnlBctAPnJSI20P81PL8BPzWpx54YXLLTm8hEsWawtcYLMrwvpK48hGxLLbCS+1KRXhsKhw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "3.2.6", + "magic-string": "^0.30.17", + "pathe": "^2.0.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/spy": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.6.tgz", + "integrity": "sha512-oq6BbH68WzcWmwtBrU9nqLeaXTR4XwJF7FSLkKEZo4i6eoXcrxjcwSuTvWBIRUTC6VC72nXYunzqgZA+IKdtxg==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyspy": "^4.0.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/utils": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.6.tgz", + "integrity": "sha512-lI23nIs4bnT3T8NIoh+vFaz5s2/DdP0Jgt2jxwgWljvwn82cLJtyi/If+fjFyoLMGIOz0U/fKvWE0d4jsNQEfg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "3.2.6", + "loupe": "^3.1.4", + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/ansis": { + "version": "4.3.1", + "resolved": "https://registry.npmjs.org/ansis/-/ansis-4.3.1.tgz", + "integrity": "sha512-BJ8/l4R5LRE7hW9WdSuGYrLSHi2ynxeFpDFbH0K/CgNeY/tyhk+vO6TYxXC5r5CpUhNVX310xzPsN/H9lCdfOA==", + "dev": true, + "license": "ISC", + "engines": { + "node": ">=14" + } + }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, + "node_modules/ast-kit": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/ast-kit/-/ast-kit-3.0.0.tgz", + "integrity": "sha512-8OG92q3R35qjC/4i6BLBMg8IB+fClWu/1PEwg2Z9Rn+BuNaiEgJzpzn+pxWOdHJWDCAwu2JP0wCDTozAM4QirQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^8.0.0", + "estree-walker": "^3.0.3", + "pathe": "^2.0.3" + }, + "engines": { + "node": "^22.18.0 || >=24.11.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + } + }, + "node_modules/birpc": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/birpc/-/birpc-4.0.0.tgz", + "integrity": "sha512-LShSxJP0KTmd101b6DRyGBj57LZxSDYWKitQNW/mi8GRMvZb078Uf9+pveax1DrVL89vm7mWe+TovdI/UDOuPw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/cac": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/cac/-/cac-7.0.0.tgz", + "integrity": "sha512-tixWYgm5ZoOD+3g6UTea91eow5z6AAHaho3g0V9CNSNb45gM8SmflpAc+GRd1InC4AqN/07Unrgp56Y94N9hJQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/chai": { + "version": "5.3.3", + "resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz", + "integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==", + "dev": true, + "license": "MIT", + "dependencies": { + "assertion-error": "^2.0.1", + "check-error": "^2.1.1", + "deep-eql": "^5.0.1", + "loupe": "^3.1.0", + "pathval": "^2.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/check-error": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz", + "integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 16" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "dev": true, + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/deep-eql": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz", + "integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/defu": { + "version": "6.1.7", + "resolved": "https://registry.npmjs.org/defu/-/defu-6.1.7.tgz", + "integrity": "sha512-7z22QmUWiQ/2d0KkdYmANbRUVABpZ9SNYyH5vx6PZ+nE5bcC0l7uFvEfHlyld/HcGBFTL536ClDt3DEcSlEJAQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/dts-resolver": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/dts-resolver/-/dts-resolver-3.0.0.tgz", + "integrity": "sha512-1T1f+z+4tl9XD+m+0HBgWoL/nm0bOIffyWaUuUSBlFg/86IWvfx+wjNaO/ybU0AJzG9/Mi5hBUgGV6zCmWEN7Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^22.18.0 || >=24.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + }, + "peerDependencies": { + "oxc-resolver": ">=11.0.0" + }, + "peerDependenciesMeta": { + "oxc-resolver": { + "optional": true + } + } + }, + "node_modules/empathic": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/empathic/-/empathic-2.0.1.tgz", + "integrity": "sha512-YGRs8knHhKHVShLkFET/rWAU8kmHbOV5LwN938RHI0pljAJ1Gf6SzXsSmRaEzcXTtOOmVqJ5+WtQPL5uigY50Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14" + } + }, + "node_modules/es-module-lexer": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz", + "integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==", + "dev": true, + "license": "MIT" + }, + "node_modules/esbuild": { + "version": "0.28.1", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.1.tgz", + "integrity": "sha512-HrJrvZv5ayxBzPfwphOoNzkzOIIlifzk0KJrGK2c8R4+LKpMtpYLQeUdjnwjWv/LZlkH2laZk+4w78pi99D4Vw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.28.1", + "@esbuild/android-arm": "0.28.1", + "@esbuild/android-arm64": "0.28.1", + "@esbuild/android-x64": "0.28.1", + "@esbuild/darwin-arm64": "0.28.1", + "@esbuild/darwin-x64": "0.28.1", + "@esbuild/freebsd-arm64": "0.28.1", + "@esbuild/freebsd-x64": "0.28.1", + "@esbuild/linux-arm": "0.28.1", + "@esbuild/linux-arm64": "0.28.1", + "@esbuild/linux-ia32": "0.28.1", + "@esbuild/linux-loong64": "0.28.1", + "@esbuild/linux-mips64el": "0.28.1", + "@esbuild/linux-ppc64": "0.28.1", + "@esbuild/linux-riscv64": "0.28.1", + "@esbuild/linux-s390x": "0.28.1", + "@esbuild/linux-x64": "0.28.1", + "@esbuild/netbsd-arm64": "0.28.1", + "@esbuild/netbsd-x64": "0.28.1", + "@esbuild/openbsd-arm64": "0.28.1", + "@esbuild/openbsd-x64": "0.28.1", + "@esbuild/openharmony-arm64": "0.28.1", + "@esbuild/sunos-x64": "0.28.1", + "@esbuild/win32-arm64": "0.28.1", + "@esbuild/win32-ia32": "0.28.1", + "@esbuild/win32-x64": "0.28.1" + } + }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, + "node_modules/expect-type": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", + "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/get-tsconfig": { + "version": "5.0.0-beta.5", + "resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-5.0.0-beta.5.tgz", + "integrity": "sha512-/6gFNr0N04nob252sTQxyFLi3eKFRqIg1I87YcqAMT1i6SQrSF6KujUEQrtrjMV0H/eejTCltLdDSTEMzHbnsQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "resolve-pkg-maps": "^1.0.0" + }, + "engines": { + "node": ">=20.20.0" + }, + "funding": { + "url": "https://github.com/privatenumber/get-tsconfig?sponsor=1" + } + }, + "node_modules/hookable": { + "version": "6.1.1", + "resolved": "https://registry.npmjs.org/hookable/-/hookable-6.1.1.tgz", + "integrity": "sha512-U9LYDy1CwhMCnprUfeAZWZGByVbhd54hwepegYTK7Pi5NvqEj63ifz5z+xukznehT7i6NIZRu89Ay1AZmRsLEQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/import-without-cache": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/import-without-cache/-/import-without-cache-0.4.0.tgz", + "integrity": "sha512-NkJQA7oZ4YHQhd2+H3BoRFKF3d/XNsiKpHZCQEMH9pDX27hQQLsTyOocyRgaIVtf8gHX3Nt3LPkR4e5EdtPAGQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^22.18.0 || >=24.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + } + }, + "node_modules/js-tokens": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz", + "integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/jsesc": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", + "integrity": "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA==", + "dev": true, + "license": "MIT", + "bin": { + "jsesc": "bin/jsesc" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/loupe": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz", + "integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "dev": true, + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.15", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.15.tgz", + "integrity": "sha512-y7Wygv/7mEOvxTuEQDB8StXdMRBWf1kR/tlhAzBRUFkB2jfcLOAxO/SHmOO2zgz1pVgK29/kyupn059/bCHdjA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/obug": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/obug/-/obug-2.1.3.tgz", + "integrity": "sha512-9miFgM2OFba7hB+pRgvtV84pYTBaoTHohvmIgiRt6dRIzbwEOIaNaP+dIlGs2fNFoB0SeISs0Jz5WFVRid6Xyg==", + "dev": true, + "funding": [ + "https://github.com/sponsors/sxzz", + "https://opencollective.com/debug" + ], + "license": "MIT", + "engines": { + "node": ">=12.20.0" + } + }, + "node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, + "node_modules/pathval": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz", + "integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.16" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/picomatch": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", + "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/postcss": { + "version": "8.5.15", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.15.tgz", + "integrity": "sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.12", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/quansync": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/quansync/-/quansync-1.0.0.tgz", + "integrity": "sha512-5xZacEEufv3HSTPQuchrvV6soaiACMFnq1H8wkVioctoH3TRha9Sz66lOxRwPK/qZj7HPiSveih9yAyh98gvqA==", + "dev": true, + "funding": [ + { + "type": "individual", + "url": "https://github.com/sponsors/antfu" + }, + { + "type": "individual", + "url": "https://github.com/sponsors/sxzz" + } + ], + "license": "MIT" + }, + "node_modules/resolve-pkg-maps": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz", + "integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1" + } + }, + "node_modules/rolldown": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.1.3.tgz", + "integrity": "sha512-1F1eEtUBtFvcGm1HQ9TiUIUHPQG7mSAODrhIzjxoUEFuo8OcbrGLiVLkevNgj84TE4lnHvnumwFjhJO5Eu135g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@oxc-project/types": "=0.137.0", + "@rolldown/pluginutils": "^1.0.0" + }, + "bin": { + "rolldown": "bin/cli.mjs" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "optionalDependencies": { + "@rolldown/binding-android-arm64": "1.1.3", + "@rolldown/binding-darwin-arm64": "1.1.3", + "@rolldown/binding-darwin-x64": "1.1.3", + "@rolldown/binding-freebsd-x64": "1.1.3", + "@rolldown/binding-linux-arm-gnueabihf": "1.1.3", + "@rolldown/binding-linux-arm64-gnu": "1.1.3", + "@rolldown/binding-linux-arm64-musl": "1.1.3", + "@rolldown/binding-linux-ppc64-gnu": "1.1.3", + "@rolldown/binding-linux-s390x-gnu": "1.1.3", + "@rolldown/binding-linux-x64-gnu": "1.1.3", + "@rolldown/binding-linux-x64-musl": "1.1.3", + "@rolldown/binding-openharmony-arm64": "1.1.3", + "@rolldown/binding-wasm32-wasi": "1.1.3", + "@rolldown/binding-win32-arm64-msvc": "1.1.3", + "@rolldown/binding-win32-x64-msvc": "1.1.3" + } + }, + "node_modules/rolldown-plugin-dts": { + "version": "0.26.0", + "resolved": "https://registry.npmjs.org/rolldown-plugin-dts/-/rolldown-plugin-dts-0.26.0.tgz", + "integrity": "sha512-e+kEPtUiDES0htk5iqkSeF4EzAV7R+vugGB44iPDuw1Kw9E+WyL1VG7PaV0IIjGHLiacztMBcMTyrr8ON9CT1Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/generator": "^8.0.0", + "@babel/helper-validator-identifier": "^8.0.0", + "@babel/parser": "^8.0.0", + "ast-kit": "^3.0.0", + "birpc": "^4.0.0", + "dts-resolver": "^3.0.0", + "get-tsconfig": "5.0.0-beta.5", + "obug": "^2.1.3" + }, + "engines": { + "node": "^22.18.0 || >=24.11.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + }, + "peerDependencies": { + "@ts-macro/tsc": "^0.3.6", + "@typescript/native-preview": ">=7.0.0-dev.20260325.1", + "rolldown": "^1.0.0", + "typescript": "^5.0.0 || ^6.0.0", + "vue-tsc": "~3.2.0 || ~3.3.0" + }, + "peerDependenciesMeta": { + "@ts-macro/tsc": { + "optional": true + }, + "@typescript/native-preview": { + "optional": true + }, + "typescript": { + "optional": true + }, + "vue-tsc": { + "optional": true + } + } + }, + "node_modules/rollup": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.62.2.tgz", + "integrity": "sha512-RFnrW4lhXA3s3eqHDZvN654g8OTjzRfqpIRJYczCGB6HzphckVAi/Qh4tbPUbRuDi7s1Llv8g/NspLkttY3gTA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.9" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@rollup/rollup-android-arm-eabi": "4.62.2", + "@rollup/rollup-android-arm64": "4.62.2", + "@rollup/rollup-darwin-arm64": "4.62.2", + "@rollup/rollup-darwin-x64": "4.62.2", + "@rollup/rollup-freebsd-arm64": "4.62.2", + "@rollup/rollup-freebsd-x64": "4.62.2", + "@rollup/rollup-linux-arm-gnueabihf": "4.62.2", + "@rollup/rollup-linux-arm-musleabihf": "4.62.2", + "@rollup/rollup-linux-arm64-gnu": "4.62.2", + "@rollup/rollup-linux-arm64-musl": "4.62.2", + "@rollup/rollup-linux-loong64-gnu": "4.62.2", + "@rollup/rollup-linux-loong64-musl": "4.62.2", + "@rollup/rollup-linux-ppc64-gnu": "4.62.2", + "@rollup/rollup-linux-ppc64-musl": "4.62.2", + "@rollup/rollup-linux-riscv64-gnu": "4.62.2", + "@rollup/rollup-linux-riscv64-musl": "4.62.2", + "@rollup/rollup-linux-s390x-gnu": "4.62.2", + "@rollup/rollup-linux-x64-gnu": "4.62.2", + "@rollup/rollup-linux-x64-musl": "4.62.2", + "@rollup/rollup-openbsd-x64": "4.62.2", + "@rollup/rollup-openharmony-arm64": "4.62.2", + "@rollup/rollup-win32-arm64-msvc": "4.62.2", + "@rollup/rollup-win32-ia32-msvc": "4.62.2", + "@rollup/rollup-win32-x64-gnu": "4.62.2", + "@rollup/rollup-win32-x64-msvc": "4.62.2", + "fsevents": "~2.3.2" + } + }, + "node_modules/semver": { + "version": "7.8.5", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", + "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", + "dev": true, + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/siginfo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==", + "dev": true, + "license": "ISC" + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/stackback": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz", + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==", + "dev": true, + "license": "MIT" + }, + "node_modules/std-env": { + "version": "3.10.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz", + "integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==", + "dev": true, + "license": "MIT" + }, + "node_modules/strip-literal": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-3.1.0.tgz", + "integrity": "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==", + "dev": true, + "license": "MIT", + "dependencies": { + "js-tokens": "^9.0.1" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/tinybench": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyexec": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.2.4.tgz", + "integrity": "sha512-SHf/r48b7vOrjve9PxJo3MN5v5yuyjHvdUcrQffT3WXMUfnGmHDVbC4k3sHJaJTgZCwpUplIaAo5ANtMyp3YHg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.4" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tinypool": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz", + "integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^18.0.0 || >=20.0.0" + } + }, + "node_modules/tinyrainbow": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz", + "integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tinyspy": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz", + "integrity": "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tree-kill": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/tree-kill/-/tree-kill-1.2.2.tgz", + "integrity": "sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==", + "dev": true, + "license": "MIT", + "bin": { + "tree-kill": "cli.js" + } + }, + "node_modules/tsdown": { + "version": "0.22.3", + "resolved": "https://registry.npmjs.org/tsdown/-/tsdown-0.22.3.tgz", + "integrity": "sha512-louqbfA8Qf//B9jTTL0FPtXTNpjCWv1VPkbcmQMph2pTpzs+LnB1tbe4tDDRVpo2BjF5SgUXaTZe45SxB8pWHg==", + "dev": true, + "license": "MIT", + "dependencies": { + "ansis": "^4.3.1", + "cac": "^7.0.0", + "defu": "^6.1.7", + "empathic": "^2.0.1", + "hookable": "^6.1.1", + "import-without-cache": "^0.4.0", + "obug": "^2.1.3", + "picomatch": "^4.0.4", + "rolldown": "~1.1.1", + "rolldown-plugin-dts": "^0.26.0", + "semver": "^7.8.4", + "tinyexec": "^1.2.4", + "tinyglobby": "^0.2.17", + "tree-kill": "^1.2.2", + "unconfig-core": "^7.5.0" + }, + "bin": { + "tsdown": "dist/run.mjs" + }, + "engines": { + "node": "^22.18.0 || >=24.11.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + }, + "peerDependencies": { + "@arethetypeswrong/core": "^0.18.1", + "@tsdown/css": "0.22.3", + "@tsdown/exe": "0.22.3", + "@vitejs/devtools": "*", + "publint": "^0.3.8", + "tsx": "*", + "typescript": "^5.0.0 || ^6.0.0", + "unplugin-unused": "^0.5.0", + "unrun": "*" + }, + "peerDependenciesMeta": { + "@arethetypeswrong/core": { + "optional": true + }, + "@tsdown/css": { + "optional": true + }, + "@tsdown/exe": { + "optional": true + }, + "@vitejs/devtools": { + "optional": true + }, + "publint": { + "optional": true + }, + "tsx": { + "optional": true + }, + "typescript": { + "optional": true + }, + "unplugin-unused": { + "optional": true + }, + "unrun": { + "optional": true + } + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "dev": true, + "license": "0BSD", + "optional": true + }, + "node_modules/typescript": { + "version": "5.9.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", + "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/unconfig-core": { + "version": "7.5.0", + "resolved": "https://registry.npmjs.org/unconfig-core/-/unconfig-core-7.5.0.tgz", + "integrity": "sha512-Su3FauozOGP44ZmKdHy2oE6LPjk51M/TRRjHv2HNCWiDvfvCoxC2lno6jevMA91MYAdCdwP05QnWdWpSbncX/w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@quansync/fs": "^1.0.0", + "quansync": "^1.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/undici-types": { + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", + "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/vite": { + "version": "7.3.6", + "resolved": "https://registry.npmjs.org/vite/-/vite-7.3.6.tgz", + "integrity": "sha512-4XP60spRGjSZFf1qYH+dJIkK2znL3zQfl9KkOV9MkkRR/3Dls0dxaBsQPTloEc5BLXWPL9vsOxopxyKoMmDueg==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.27.0 || ^0.28.0", + "fdir": "^6.5.0", + "picomatch": "^4.0.3", + "postcss": "^8.5.6", + "rollup": "^4.43.0", + "tinyglobby": "^0.2.15" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^20.19.0 || >=22.12.0", + "jiti": ">=1.21.0", + "less": "^4.0.0", + "lightningcss": "^1.21.0", + "sass": "^1.70.0", + "sass-embedded": "^1.70.0", + "stylus": ">=0.54.8", + "sugarss": "^5.0.0", + "terser": "^5.16.0", + "tsx": "^4.8.1", + "yaml": "^2.4.2" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "jiti": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + }, + "node_modules/vite-node": { + "version": "3.2.4", + "resolved": "https://registry.npmjs.org/vite-node/-/vite-node-3.2.4.tgz", + "integrity": "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==", + "dev": true, + "license": "MIT", + "dependencies": { + "cac": "^6.7.14", + "debug": "^4.4.1", + "es-module-lexer": "^1.7.0", + "pathe": "^2.0.3", + "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" + }, + "bin": { + "vite-node": "vite-node.mjs" + }, + "engines": { + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/vite-node/node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/vitest": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.6.tgz", + "integrity": "sha512-xejya+bT/j/+R/AGa1XOfRxLmNUlLtlwjRsFUILF+xHfzElmGcmFydy2gqqIrd62ptIEfwVMofd19uNWD9L7Nw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/chai": "^5.2.2", + "@vitest/expect": "3.2.6", + "@vitest/mocker": "3.2.6", + "@vitest/pretty-format": "^3.2.6", + "@vitest/runner": "3.2.6", + "@vitest/snapshot": "3.2.6", + "@vitest/spy": "3.2.6", + "@vitest/utils": "3.2.6", + "chai": "^5.2.0", + "debug": "^4.4.1", + "expect-type": "^1.2.1", + "magic-string": "^0.30.17", + "pathe": "^2.0.3", + "picomatch": "^4.0.2", + "std-env": "^3.9.0", + "tinybench": "^2.9.0", + "tinyexec": "^0.3.2", + "tinyglobby": "^0.2.14", + "tinypool": "^1.1.1", + "tinyrainbow": "^2.0.0", + "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0", + "vite-node": "3.2.4", + "why-is-node-running": "^2.3.0" + }, + "bin": { + "vitest": "vitest.mjs" + }, + "engines": { + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@edge-runtime/vm": "*", + "@types/debug": "^4.1.12", + "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "@vitest/browser": "3.2.6", + "@vitest/ui": "3.2.6", + "happy-dom": "*", + "jsdom": "*" + }, + "peerDependenciesMeta": { + "@edge-runtime/vm": { + "optional": true + }, + "@types/debug": { + "optional": true + }, + "@types/node": { + "optional": true + }, + "@vitest/browser": { + "optional": true + }, + "@vitest/ui": { + "optional": true + }, + "happy-dom": { + "optional": true + }, + "jsdom": { + "optional": true + } + } + }, + "node_modules/vitest/node_modules/tinyexec": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz", + "integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==", + "dev": true, + "license": "MIT" + }, + "node_modules/why-is-node-running": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "siginfo": "^2.0.0", + "stackback": "0.0.2" + }, + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=8" + } + } + } +} diff --git a/benchmarks/adapters/online-mind2web/judge/package.json b/benchmarks/adapters/online-mind2web/judge/package.json new file mode 100644 index 0000000..44890b9 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/package.json @@ -0,0 +1,25 @@ +{ + "name": "online-mind2web-webjudge", + "version": "0.1.0", + "description": "Self-contained WebJudge verifier bin for the Online-Mind2Web Harbor adapter; runs in-VM via node, grades the agent trajectory against an Anthropic judge", + "license": "MIT", + "private": true, + "type": "module", + "bin": { + "online-mind2web-webjudge": "./dist/judge.js" + }, + "scripts": { + "build": "tsdown && chmod +x dist/judge.js", + "typecheck": "tsc --noEmit", + "test": "vitest --run" + }, + "engines": { + "node": ">=22.0.0" + }, + "devDependencies": { + "@types/node": "^22.10.0", + "tsdown": "^0.22.2", + "typescript": "5.9.3", + "vitest": "^3.2.4" + } +} diff --git a/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts b/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts new file mode 100644 index 0000000..278dac9 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts @@ -0,0 +1,102 @@ +import { existsSync, readFileSync } from "node:fs"; +import { dirname, isAbsolute, join } from "node:path"; +import type { BenchTask, Trajectory, TrajectoryStep } from "./types.ts"; + +/** Grader input written by the adapter (tests/task.json). */ +interface TaskJson { + task_id: string; + instruction: string; + start_url?: string | null; + reference_length?: number | null; +} + +const MIME_BY_EXT: Record = { + png: "image/png", + webp: "image/webp", + jpeg: "image/jpeg", + jpg: "image/jpeg", + gif: "image/gif", +}; + +function mimeForPath(path: string): string { + const ext = path.split(".").pop()?.toLowerCase() ?? "png"; + return MIME_BY_EXT[ext] ?? "image/png"; +} + +export function loadTask(taskJsonPath: string): BenchTask { + const raw = JSON.parse(readFileSync(taskJsonPath, "utf8")) as TaskJson; + return { + id: raw.task_id, + instruction: raw.instruction, + startUrl: raw.start_url ?? undefined, + metadata: + raw.reference_length != null ? { referenceLength: raw.reference_length } : undefined, + }; +} + +/** A description of one tool call, used as the WebJudge "action" string. */ +function actionString(name: string, args: unknown): string { + if (args && typeof args === "object" && Object.keys(args).length > 0) { + return `${name} ${JSON.stringify(args)}`; + } + return name; +} + +/** + * Reconstruct a WebJudge {@link Trajectory} from the shared-core artifacts. + * + * The Node entrypoint (`benchmarks/node/src/sink.ts`) writes `run.jsonl` with + * `assistant` lines carrying `tool_calls: [{id, name, arguments}]` and + * `tool_result` lines carrying `{call_id, shots: ["shots/shot-N.png"]}`. Each + * spilled screenshot becomes one step whose action is the tool call that + * produced it; the bytes are read from disk into base64 for the judge. The + * final answer comes from `answer.txt` (the grading channel), falling back to + * the `final` record. + */ +export function loadTrajectory(opts: { + runJsonlPath: string; + answerPath: string; + shotsBaseDir?: string; +}): Trajectory { + const baseDir = opts.shotsBaseDir ?? dirname(opts.answerPath); + + let finalAnswer = ""; + if (existsSync(opts.answerPath)) { + finalAnswer = readFileSync(opts.answerPath, "utf8").trim(); + } + + const steps: TrajectoryStep[] = []; + if (existsSync(opts.runJsonlPath)) { + const callActions = new Map(); + const lines = readFileSync(opts.runJsonlPath, "utf8").split("\n"); + for (const line of lines) { + const trimmed = line.trim(); + if (!trimmed) continue; + const rec = JSON.parse(trimmed) as Record; + if (rec.t === "assistant") { + for (const tc of (rec.tool_calls as Array>) ?? []) { + callActions.set( + String(tc.id), + actionString(String(tc.name ?? ""), tc.arguments), + ); + } + } else if (rec.t === "tool_result") { + const action = callActions.get(String(rec.call_id)) ?? "tool_result"; + for (const rel of (rec.shots as string[]) ?? []) { + const abs = isAbsolute(rel) ? rel : join(baseDir, rel); + if (!existsSync(abs)) continue; + steps.push({ + index: steps.length, + action, + screenshotBase64: readFileSync(abs).toString("base64"), + screenshotMimeType: mimeForPath(rel), + }); + } + } else if (rec.t === "final" && !finalAnswer) { + finalAnswer = String(rec.answer ?? ""); + } + } + } + + return { steps, finalAnswer }; +} diff --git a/benchmarks/adapters/online-mind2web/judge/src/judge.ts b/benchmarks/adapters/online-mind2web/judge/src/judge.ts new file mode 100644 index 0000000..293d02c --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/src/judge.ts @@ -0,0 +1,97 @@ +/** + * WebJudge verifier entrypoint for the Online-Mind2Web Harbor adapter. + * + * Runs inside the Kernel browser VM (which ships `node` + global `fetch`) as a + * self-contained bundle — no `npm install` at verify time. Reads the agent's + * artifacts under `/logs/agent` (answer.txt + run.jsonl + shots/), reconstructs + * the WebJudge trajectory, grades it with an Anthropic-backed judge, and writes + * a single reward float to `/logs/verifier/reward.txt`. + * + * A missing/empty trajectory or any failure writes reward `0` rather than + * leaving the reward file empty (an empty reward is a Harbor verifier error). + */ +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; +import { loadTask, loadTrajectory } from "./artifacts.ts"; +import { anthropicJudgeModel } from "./model.ts"; +import { gradeWithWebJudge } from "./webjudge.ts"; + +interface Args { + task: string; + run: string; + answer: string; + shots?: string; + judgeModel: string; + scoreThreshold: number; + rewardOut: string; + detailsOut?: string; +} + +function parseArgs(argv: string[]): Args { + const flags = new Map(); + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + if (arg.startsWith("--")) { + flags.set(arg.slice(2), argv[i + 1] ?? ""); + i += 1; + } + } + const require = (name: string): string => { + const value = flags.get(name); + if (!value) throw new Error(`missing required --${name}`); + return value; + }; + return { + task: require("task"), + run: require("run"), + answer: require("answer"), + shots: flags.get("shots"), + judgeModel: flags.get("judge-model") ?? "anthropic:claude-sonnet-4-6", + scoreThreshold: Number(flags.get("score-threshold") ?? "3"), + rewardOut: require("reward-out"), + detailsOut: flags.get("details-out"), + }; +} + +function writeReward(path: string, reward: 0 | 1): void { + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, `${reward}\n`); +} + +async function main(): Promise { + const args = parseArgs(process.argv.slice(2)); + try { + const task = loadTask(args.task); + const trajectory = loadTrajectory({ + runJsonlPath: args.run, + answerPath: args.answer, + shotsBaseDir: args.shots, + }); + const judge = anthropicJudgeModel(args.judgeModel); + const result = await gradeWithWebJudge({ + task, + trajectory, + judge, + scoreThreshold: args.scoreThreshold, + }); + writeReward(args.rewardOut, result.success ? 1 : 0); + if (args.detailsOut) { + mkdirSync(dirname(args.detailsOut), { recursive: true }); + writeFileSync( + args.detailsOut, + JSON.stringify( + { success: result.success, reasoning: result.reasoning, details: result.details }, + null, + 2, + ), + ); + } + } catch (err) { + // Never leave the reward file empty: a grading failure is a 0, and the + // error is surfaced on stderr for the verifier log. + console.error(err instanceof Error ? (err.stack ?? err.message) : String(err)); + writeReward(args.rewardOut, 0); + } +} + +main(); diff --git a/benchmarks/adapters/online-mind2web/judge/src/model.ts b/benchmarks/adapters/online-mind2web/judge/src/model.ts new file mode 100644 index 0000000..19a631d --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/src/model.ts @@ -0,0 +1,75 @@ +import type { JudgeContent, JudgeModel } from "./types.ts"; + +/** + * A {@link JudgeModel} backed by the Anthropic Messages API over `fetch`. + * + * Self-contained so the bundled judge runs inside the Kernel browser VM with no + * `npm install` at verify time (the VM ships `node` and global `fetch` but not + * the cua SDKs). The judge model is given as a `provider:name` ref; only the + * `anthropic` provider is wired because it is the configured judge backbone. + */ +const ANTHROPIC_URL = "https://api.anthropic.com/v1/messages"; +const ANTHROPIC_VERSION = "2023-06-01"; + +interface AnthropicBlock { + type: string; + text?: string; +} + +export function parseModelRef(ref: string): { provider: string; name: string } { + const idx = ref.indexOf(":"); + if (idx === -1) return { provider: "anthropic", name: ref }; + return { provider: ref.slice(0, idx), name: ref.slice(idx + 1) }; +} + +function toAnthropicContent(content: JudgeContent): unknown[] { + return content.map((part) => + part.type === "text" + ? { type: "text", text: part.text } + : { + type: "image", + source: { type: "base64", media_type: part.mimeType, data: part.data }, + }, + ); +} + +/** Anthropic-backed judge. Resolves the API key from `ANTHROPIC_API_KEY`. */ +export function anthropicJudgeModel(ref: string): JudgeModel { + const { provider, name } = parseModelRef(ref); + if (provider !== "anthropic") { + throw new Error( + `unsupported judge provider "${provider}" (only anthropic is wired); got "${ref}"`, + ); + } + const apiKey = process.env.ANTHROPIC_API_KEY; + if (!apiKey) { + throw new Error("ANTHROPIC_API_KEY is required for the WebJudge model"); + } + return { + async complete(systemPrompt, content) { + const res = await fetch(ANTHROPIC_URL, { + method: "POST", + headers: { + "content-type": "application/json", + "x-api-key": apiKey, + "anthropic-version": ANTHROPIC_VERSION, + }, + body: JSON.stringify({ + model: name, + max_tokens: 1024, + temperature: 0, + system: systemPrompt, + messages: [{ role: "user", content: toAnthropicContent(content) }], + }), + }); + if (!res.ok) { + const body = await res.text(); + throw new Error(`Anthropic API error ${res.status}: ${body}`); + } + const data = (await res.json()) as { content?: AnthropicBlock[] }; + return (data.content ?? []) + .flatMap((b) => (b.type === "text" && b.text ? [b.text] : [])) + .join(""); + }, + }; +} diff --git a/benchmarks/adapters/online-mind2web/judge/src/prompts.ts b/benchmarks/adapters/online-mind2web/judge/src/prompts.ts new file mode 100644 index 0000000..d4b6684 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/src/prompts.ts @@ -0,0 +1,139 @@ +/** + * WebJudge prompts and parsers, ported from + * OSU-NLP-Group/Online-Mind2Web `src/methods/webjudge_online_mind2web.py`. + * + * The prompt text and the literal markers (`**Key Points**`, `Score`, `Status:`) + * are reproduced verbatim because the parsers below depend on them. + */ + +export const MAX_IMAGE = 50; + +export const KEY_POINTS_SYSTEM = `You are an expert tasked with analyzing a given task to identify the key points explicitly stated in the task description. + +**Objective**: Carefully analyze the task description and extract the critical elements explicitly mentioned in the task for achieving its goal. + +**Instructions**: +1. Read the task description carefully. +2. Identify and extract **key points** directly stated in the task description. + - A **key point** is a critical element, condition, or step explicitly mentioned in the task description. + - Do not infer or add any unstated elements. + - Words such as "best," "highest," "cheapest," "latest," "most recent," "lowest," "closest," "highest-rated," "largest," and "newest" must go through the sort function(e.g., the key point should be "Filter by highest"). + +**Respond with**: +- **Key Points**: A numbered list of the explicit key points for completing this task, one per line, without explanations or additional details.`; + +export const JUDGE_IMAGE_SYSTEM = `You are an expert evaluator tasked with determining whether an image contains information about the necessary steps to complete a task. + +**Objective**: Analyze the provided image and decide if it shows essential steps or evidence required for completing the task. Use your reasoning to explain your decision before assigning a score. + +**Instructions**: +1. Provide a detailed description of the image, including its contents, visible elements, text (if any), and any notable features. + +2. Carefully examine the image and evaluate whether it contains necessary steps or evidence crucial to task completion: +- Identify key points that could be relevant to task completion, such as actions, progress indicators, tool usage, applied filters, or step-by-step instructions. +- Does the image show actions, progress indicators, or critical information directly related to completing the task? +- Is this information indispensable for understanding or ensuring task success? +- If the image contains partial but relevant information, consider its usefulness rather than dismissing it outright. + +3. Provide your response in the following format: +- **Reasoning**: Explain your thought process and observations. Mention specific elements in the image that indicate necessary steps, evidence, or lack thereof. +- **Score**: Assign a score based on the reasoning, using the following scale: + - **1**: The image does not contain any necessary steps or relevant information. + - **2**: The image contains minimal or ambiguous information, unlikely to be essential. + - **3**: The image includes some relevant steps or hints but lacks clarity or completeness. + - **4**: The image contains important steps or evidence that are highly relevant but not fully comprehensive. + - **5**: The image clearly displays necessary steps or evidence crucial for completing the task. + +Respond with: +1. **Reasoning**: [Your explanation] +2. **Score**: [1-5]`; + +export const FINAL_JUDGE_SYSTEM = `You are an expert in evaluating the performance of a web navigation agent. The agent is designed to help a human user navigate a website to complete a task. Given the user's task, the agent's action history, key points for task completion, some potentially important web pages in the agent's trajectory and their reasons, your goal is to determine whether the agent has completed the task and achieved all requirements. + +Your response must strictly follow the following evaluation criteria! +*Important Evaluation Criteria*: +1: The filtered results must be displayed correctly. If filters were not properly applied (i.e., missing selection, missing confirmation, or no visible effect in results), the task is not considered successful. +2: You must carefully check whether these snapshots and action history meet these key points. Ensure that specific filter conditions, such as "best," "highest," "cheapest," "latest," "most recent," "lowest," "closest," "highest-rated," "largest," and "newest" are correctly applied using the filter function(e.g., sort function). +3: Certain key points or requirements should be applied by the filter. Otherwise, a search with all requirements as input will be deemed a failure since it cannot guarantee that all results meet the requirements! +4: If the task requires filtering by a specific range of money, years, or the number of beds and bathrooms, the applied filter must exactly match the given requirement. Any deviation results in failure. To ensure the task is successful, the applied filter must precisely match the specified range without being too broad or too narrow. +Examples of Failure Cases: +- If the requirement is less than $50, but the applied filter is less than $25, it is a failure. +- If the requirement is $1500-$2500, but the applied filter is $2000-$2500, it is a failure. +- If the requirement is $25-$200, but the applied filter is $0-$200, it is a failure. +- If the required years are 2004-2012, but the filter applied is 2001-2012, it is a failure. +- If the required years are before 2015, but the applied filter is 2000-2014, it is a failure. +- If the task requires exactly 2 beds, but the filter applied is 2+ beds, it is a failure. +5: Some tasks require a submission action or a display of results to be considered successful. +6: If the retrieved information is invalid or empty(e.g., No match was found), but the agent has correctly performed the required action, it should still be considered successful. +7: If the current page already displays all available items, then applying a filter is not necessary. As long as the agent selects items that meet the requirements (e.g., the cheapest or lowest price), the task is still considered successful. + +*IMPORTANT* +Format your response into two lines as shown below: + +Thoughts: +Status: "success" or "failure"`; + +export function keyPointsUserText(task: string): string { + return `Task: ${task}`; +} + +export function judgeImageUserText(task: string, keyPoints: string): string { + return `**Task**: ${task} + +**Key Points for Task Completion**: ${keyPoints} + +The snapshot of the web page is shown in the image.`; +} + +export function finalUserText(args: { + task: string; + keyPoints: string; + actions: string[]; + thoughts: string[]; + hasImages: boolean; +}): string { + const lastActions = args.actions.map((a, i) => `${i + 1}. ${a}`).join("\n"); + const head = `User Task: ${args.task} + +Key Points: ${args.keyPoints} + +Action History: +${lastActions}`; + if (!args.hasImages) return head; + const thoughts = args.thoughts.map((t, i) => `${i + 1}. ${t}`).join("\n"); + return `${head} + +The potentially important snapshots of the webpage in the agent's trajectory and their reasons: +${thoughts}`; +} + +export function extractKeyPoints(raw: string): string { + const collapsed = raw.replace(/\n\n/g, "\n"); + const afterMarker = collapsed.includes("**Key Points**:") + ? collapsed.split("**Key Points**:")[1] + : collapsed.split("Key Points:").at(-1); + return (afterMarker ?? "") + .split("\n") + .map((line) => line.replace(/^\s+/, "")) + .join("\n"); +} + +export function parseImageScore(raw: string): { score: number; thought: string } { + try { + const scoreText = raw.split("Score")[1] ?? ""; + const match = scoreText.match(/[1-5]/); + const score = match ? Number(match[0]) : 0; + const thought = (raw.split("**Reasoning**:").at(-1) ?? "") + .trim() + .replace(/^\n+/, "") + .split("\n\n")[0] + .replace(/\n/g, " "); + return { score, thought }; + } catch { + return { score: 0, thought: "" }; + } +} + +export function parseVerdict(raw: string): boolean { + return raw.toLowerCase().split("status:")[1]?.includes("success") ?? false; +} diff --git a/benchmarks/adapters/online-mind2web/judge/src/types.ts b/benchmarks/adapters/online-mind2web/judge/src/types.ts new file mode 100644 index 0000000..f9e3072 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/src/types.ts @@ -0,0 +1,46 @@ +/** A single benchmark task: a natural-language instruction with an optional start URL. */ +export interface BenchTask { + id: string; + instruction: string; + startUrl?: string; + metadata?: Record; +} + +/** One step of an agent run, capturing the action and the screenshot it produced. */ +export interface TrajectoryStep { + index: number; + action: string; + screenshotBase64?: string; + screenshotMimeType?: string; +} + +/** The full agent run for a task: ordered steps plus the final assistant answer. */ +export interface Trajectory { + steps: TrajectoryStep[]; + finalAnswer: string; +} + +/** Outcome of grading a trajectory against its task. */ +export interface GradeResult { + success: boolean; + reasoning: string; + details?: Record; +} + +/** Multimodal content passed to a {@link JudgeModel}. */ +export type JudgeContent = Array< + | { type: "text"; text: string } + | { type: "image"; data: string; mimeType: string } +>; + +/** A model used by a grader. Implemented by a provider call; mocked in tests. */ +export interface JudgeModel { + complete(systemPrompt: string, content: JudgeContent): Promise; +} + +export interface GradeArgs { + task: BenchTask; + trajectory: Trajectory; + judge: JudgeModel; + scoreThreshold: number; +} diff --git a/benchmarks/adapters/online-mind2web/judge/src/webjudge.ts b/benchmarks/adapters/online-mind2web/judge/src/webjudge.ts new file mode 100644 index 0000000..d7c8bfc --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/src/webjudge.ts @@ -0,0 +1,66 @@ +import * as P from "./prompts.ts"; +import type { GradeArgs, GradeResult, JudgeContent } from "./types.ts"; + +/** + * WebJudge grading pipeline ported from Online-Mind2Web: identify key points, + * score each screenshot, keep the relevant ones, then ask for a final verdict. + */ +export async function gradeWithWebJudge({ + task, + trajectory, + judge, + scoreThreshold, +}: GradeArgs): Promise { + const kpRaw = await judge.complete(P.KEY_POINTS_SYSTEM, [ + { type: "text", text: P.keyPointsUserText(task.instruction) }, + ]); + const keyPoints = P.extractKeyPoints(kpRaw); + + const shots = trajectory.steps.filter((s) => s.screenshotBase64); + const scored = await Promise.all( + shots.map(async (step) => { + const raw = await judge.complete(P.JUDGE_IMAGE_SYSTEM, [ + { type: "text", text: P.judgeImageUserText(task.instruction, keyPoints) }, + { + type: "image", + data: step.screenshotBase64!, + mimeType: step.screenshotMimeType ?? "image/png", + }, + ]); + const { score, thought } = P.parseImageScore(raw); + return { step, score, thought, raw }; + }), + ); + + const kept = scored.filter((r) => r.score >= scoreThreshold).slice(0, P.MAX_IMAGE); + const keptThoughts = kept.map((r) => r.thought).filter((t) => t !== "").slice(0, P.MAX_IMAGE); + const actions = trajectory.steps.map((s) => s.action); + + const finalContent: JudgeContent = [ + { + type: "text", + text: P.finalUserText({ + task: task.instruction, + keyPoints, + actions, + thoughts: keptThoughts, + hasImages: kept.length > 0, + }), + }, + ...kept.map((r) => ({ + type: "image" as const, + data: r.step.screenshotBase64!, + mimeType: r.step.screenshotMimeType ?? "image/png", + })), + ]; + const finalRaw = await judge.complete(P.FINAL_JUDGE_SYSTEM, finalContent); + + return { + success: P.parseVerdict(finalRaw), + reasoning: finalRaw, + details: { + keyPoints, + imageRecords: scored.map((r) => ({ score: r.score, response: r.raw })), + }, + }; +} diff --git a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts new file mode 100644 index 0000000..dc86474 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts @@ -0,0 +1,132 @@ +import { mkdtempSync, writeFileSync } from "node:fs"; +import { mkdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { loadTask, loadTrajectory } from "../src/artifacts.ts"; +import { parseModelRef } from "../src/model.ts"; + +// 1x1 transparent PNG, the same fixture the shared-core sink test uses. +const PNG_B64 = + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="; + +/** Lay out a /logs/agent dir exactly as benchmarks/node/src/sink.ts writes it. */ +function writeAgentDir(records: Array>, answer: string): string { + const dir = mkdtempSync(join(tmpdir(), "om2w-judge-")); + mkdirSync(join(dir, "shots"), { recursive: true }); + writeFileSync(join(dir, "shots", "shot-1.png"), Buffer.from(PNG_B64, "base64")); + writeFileSync( + join(dir, "run.jsonl"), + `${records.map((r) => JSON.stringify(r)).join("\n")}\n`, + ); + writeFileSync(join(dir, "answer.txt"), answer); + return dir; +} + +describe("loadTask", () => { + it("maps task.json into a BenchTask", () => { + const path = join(mkdtempSync(join(tmpdir(), "om2w-task-")), "task.json"); + writeFileSync( + path, + JSON.stringify({ + task_id: "abc_123", + instruction: "Find the cheapest flight", + start_url: "https://example.com", + reference_length: 6, + }), + ); + expect(loadTask(path)).toEqual({ + id: "abc_123", + instruction: "Find the cheapest flight", + startUrl: "https://example.com", + metadata: { referenceLength: 6 }, + }); + }); + + it("omits startUrl when start_url is null", () => { + const path = join(mkdtempSync(join(tmpdir(), "om2w-task-")), "task.json"); + writeFileSync( + path, + JSON.stringify({ task_id: "t", instruction: "do it", start_url: null, reference_length: null }), + ); + const task = loadTask(path); + expect(task.startUrl).toBeUndefined(); + expect(task.metadata).toBeUndefined(); + }); +}); + +describe("loadTrajectory", () => { + it("pairs each spilled screenshot with the tool call that produced it", () => { + const dir = writeAgentDir( + [ + { t: "user", text: "go to example.com" }, + { + t: "assistant", + tool_calls: [{ id: "call_1", name: "click", arguments: { x: 1, y: 2 } }], + }, + { t: "tool_result", call_id: "call_1", shots: ["shots/shot-1.png"] }, + { t: "final", answer: "ignored because answer.txt wins" }, + ], + "Example Domain", + ); + + const trajectory = loadTrajectory({ + runJsonlPath: join(dir, "run.jsonl"), + answerPath: join(dir, "answer.txt"), + }); + + expect(trajectory.finalAnswer).toBe("Example Domain"); + expect(trajectory.steps).toHaveLength(1); + expect(trajectory.steps[0]).toMatchObject({ + index: 0, + action: 'click {"x":1,"y":2}', + screenshotBase64: PNG_B64, + screenshotMimeType: "image/png", + }); + }); + + it("falls back to the final record when answer.txt is empty", () => { + const dir = writeAgentDir( + [ + { t: "assistant", tool_calls: [{ id: "c", name: "screenshot", arguments: {} }] }, + { t: "tool_result", call_id: "c", shots: ["shots/shot-1.png"] }, + { t: "final", answer: "from final line" }, + ], + "", + ); + + const trajectory = loadTrajectory({ + runJsonlPath: join(dir, "run.jsonl"), + answerPath: join(dir, "answer.txt"), + }); + + expect(trajectory.finalAnswer).toBe("from final line"); + expect(trajectory.steps[0].action).toBe("screenshot"); + }); + + it("returns an empty trajectory when run.jsonl is absent", () => { + const dir = mkdtempSync(join(tmpdir(), "om2w-empty-")); + const trajectory = loadTrajectory({ + runJsonlPath: join(dir, "run.jsonl"), + answerPath: join(dir, "answer.txt"), + }); + expect(trajectory.steps).toEqual([]); + expect(trajectory.finalAnswer).toBe(""); + }); +}); + +describe("parseModelRef", () => { + it("splits provider:name", () => { + expect(parseModelRef("anthropic:claude-sonnet-4-6")).toEqual({ + provider: "anthropic", + name: "claude-sonnet-4-6", + }); + }); + + it("defaults provider to anthropic when no colon", () => { + expect(parseModelRef("claude-sonnet-4-6")).toEqual({ + provider: "anthropic", + name: "claude-sonnet-4-6", + }); + }); +}); diff --git a/benchmarks/adapters/online-mind2web/judge/test/webjudge.test.ts b/benchmarks/adapters/online-mind2web/judge/test/webjudge.test.ts new file mode 100644 index 0000000..d843e5a --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/test/webjudge.test.ts @@ -0,0 +1,119 @@ +import { describe, expect, it } from "vitest"; +import { + extractKeyPoints, + FINAL_JUDGE_SYSTEM, + JUDGE_IMAGE_SYSTEM, + KEY_POINTS_SYSTEM, + parseImageScore, + parseVerdict, +} from "../src/prompts.ts"; +import type { JudgeContent, JudgeModel, Trajectory } from "../src/types.ts"; +import { gradeWithWebJudge } from "../src/webjudge.ts"; + +interface Call { + systemPrompt: string; + content: JudgeContent; +} + +/** A scripted JudgeModel: returns `responses[i]` for the i-th call and records every call. */ +function scriptedJudge(responses: string[]): { judge: JudgeModel; calls: Call[] } { + const calls: Call[] = []; + const judge: JudgeModel = { + async complete(systemPrompt, content) { + calls.push({ systemPrompt, content }); + return responses[calls.length - 1] ?? ""; + }, + }; + return { judge, calls }; +} + +function imageBlocks(content: JudgeContent): JudgeContent { + return content.filter((c) => c.type === "image"); +} + +function textBlock(content: JudgeContent): string { + const text = content.find((c) => c.type === "text"); + return text && text.type === "text" ? text.text : ""; +} + +const task = { id: "t1", instruction: "Find the cheapest flight" }; +const trajectory: Trajectory = { + steps: [ + { index: 0, action: "goto example.com", screenshotBase64: "AAAA", screenshotMimeType: "image/png" }, + { index: 1, action: "click {x:1}", screenshotBase64: "BBBB", screenshotMimeType: "image/png" }, + ], + finalAnswer: "done", +}; + +const KEY_POINTS_RESPONSE = "**Key Points**:\n1. Find cheapest flight\n2. Filter by lowest"; +const SHOT_PASS = "**Reasoning**: shows the filter applied\n\nScore: 4"; +const SHOT_FAIL = "**Reasoning**: irrelevant\nScore: 1"; + +describe("WebJudge parsers", () => { + it("extractKeyPoints strips the marker and per-line indentation", () => { + expect(extractKeyPoints(KEY_POINTS_RESPONSE)).toBe("\n1. Find cheapest flight\n2. Filter by lowest"); + }); + + it("parseImageScore reads the score and reasoning", () => { + expect(parseImageScore(SHOT_PASS)).toEqual({ score: 4, thought: "shows the filter applied" }); + expect(parseImageScore(SHOT_FAIL).score).toBe(1); + }); + + it("parseImageScore returns 0 when no score is present", () => { + expect(parseImageScore("no score here").score).toBe(0); + }); + + it("parseVerdict reads the status line", () => { + expect(parseVerdict("Thoughts: ok\nStatus: success")).toBe(true); + expect(parseVerdict("Thoughts: no\nStatus: failure")).toBe(false); + expect(parseVerdict("garbage")).toBe(false); + }); +}); + +describe("gradeWithWebJudge", () => { + it("scores screenshots, keeps those above the threshold, and returns the verdict", async () => { + const { judge, calls } = scriptedJudge([ + KEY_POINTS_RESPONSE, + SHOT_PASS, + SHOT_FAIL, + "Thoughts: looks complete\nStatus: success", + ]); + + const result = await gradeWithWebJudge({ task, trajectory, judge, scoreThreshold: 3 }); + + expect(result.success).toBe(true); + expect(result.reasoning).toContain("Status: success"); + expect(result.details?.keyPoints).toBe("\n1. Find cheapest flight\n2. Filter by lowest"); + + expect(calls[0].systemPrompt).toBe(KEY_POINTS_SYSTEM); + expect(calls[1].systemPrompt).toBe(JUDGE_IMAGE_SYSTEM); + expect(calls[1].content).toEqual([ + { type: "text", text: expect.stringContaining("**Task**: Find the cheapest flight") }, + { type: "image", data: "AAAA", mimeType: "image/png" }, + ]); + + const finalCall = calls.at(-1)!; + expect(finalCall.systemPrompt).toBe(FINAL_JUDGE_SYSTEM); + expect(imageBlocks(finalCall.content)).toHaveLength(1); + expect(imageBlocks(finalCall.content)[0]).toMatchObject({ data: "AAAA" }); + expect(textBlock(finalCall.content)).toContain("important snapshots"); + expect(textBlock(finalCall.content)).toContain("1. goto example.com"); + }); + + it("omits the snapshots section when no screenshot passes the threshold", async () => { + const { judge, calls } = scriptedJudge([ + KEY_POINTS_RESPONSE, + SHOT_FAIL, + SHOT_FAIL, + "Thoughts: nothing useful\nStatus: failure", + ]); + + const result = await gradeWithWebJudge({ task, trajectory, judge, scoreThreshold: 5 }); + + expect(result.success).toBe(false); + const finalCall = calls.at(-1)!; + expect(imageBlocks(finalCall.content)).toHaveLength(0); + expect(textBlock(finalCall.content)).not.toContain("important snapshots"); + expect(textBlock(finalCall.content)).toContain("Action History:"); + }); +}); diff --git a/benchmarks/adapters/online-mind2web/judge/tsconfig.json b/benchmarks/adapters/online-mind2web/judge/tsconfig.json new file mode 100644 index 0000000..2ea517d --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ESNext", + "moduleResolution": "Bundler", + "allowImportingTsExtensions": true, + "lib": ["ES2022"], + "strict": true, + "esModuleInterop": true, + "allowSyntheticDefaultImports": true, + "resolveJsonModule": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "noEmit": true + }, + "include": ["src/**/*.ts", "test/**/*.ts"] +} diff --git a/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts b/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts new file mode 100644 index 0000000..ad32eab --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts @@ -0,0 +1,15 @@ +import { defineConfig } from "tsdown"; + +// Bundle to a single self-contained ESM file so `node dist/judge.js` runs inside +// the Kernel VM with no dependency install. No externals: the judge only uses +// node builtins and global fetch. +export default defineConfig({ + entry: ["src/judge.ts"], + format: ["esm"], + platform: "node", + dts: false, + sourcemap: false, + clean: true, + noExternal: [/.*/], + outExtensions: () => ({ js: ".js" }), +}); diff --git a/benchmarks/adapters/online-mind2web/judge/vitest.config.ts b/benchmarks/adapters/online-mind2web/judge/vitest.config.ts new file mode 100644 index 0000000..6ec3464 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/vitest.config.ts @@ -0,0 +1,12 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + server: { + host: "127.0.0.1", + }, + test: { + globals: true, + environment: "node", + testTimeout: 30000, + }, +}); diff --git a/benchmarks/adapters/online-mind2web/pyproject.toml b/benchmarks/adapters/online-mind2web/pyproject.toml new file mode 100644 index 0000000..f778be9 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/pyproject.toml @@ -0,0 +1,28 @@ +[project] +name = "harbor-online-mind2web-adapter" +version = "0.1.0" +description = "Harbor adapter for Online-Mind2Web (Kernel env + cua agent, WebJudge grader)" +readme = "README.md" +authors = [{ name = "Kernel" }] +requires-python = ">=3.10" +dependencies = [] + +[project.scripts] +online-mind2web = "online_mind2web.main:main" + +[build-system] +requires = ["uv_build>=0.9.18,<0.10.0"] +build-backend = "uv_build" + +[tool.uv.build-backend] +module-name = "online_mind2web" +module-root = "src" + +[dependency-groups] +dev = ["pytest>=8", "ruff>=0.6"] + +[tool.ruff] +target-version = "py310" + +[tool.pytest.ini_options] +testpaths = ["tests"] diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/__init__.py b/benchmarks/adapters/online-mind2web/src/online_mind2web/__init__.py new file mode 100644 index 0000000..9a75601 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/__init__.py @@ -0,0 +1,5 @@ +"""Online-Mind2Web Harbor adapter: dataset -> Kernel task dirs, graded by WebJudge.""" + +from online_mind2web.adapter import OnlineMind2WebAdapter, OnlineMind2WebTask + +__all__ = ["OnlineMind2WebAdapter", "OnlineMind2WebTask"] diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py b/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py new file mode 100644 index 0000000..0d581fd --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py @@ -0,0 +1,245 @@ +"""Online-Mind2Web adapter — converts the OSU NLP Online-Mind2Web dataset into +Harbor task dirs that run on the Kernel environment and are graded by WebJudge. + +Online-Mind2Web is 300 live web tasks (osunlp/Online-Mind2Web). Each row is a +``(instruction, start_url)`` pair with no gold answer; grading is trajectory +based (the WebJudge Node bin under ``judge/``). The adapter mirrors the field +mapping and skip-malformed logic of the cua TypeScript loader so generated task +ids stay aligned with the upstream set. + +Source: https://huggingface.co/datasets/osunlp/Online-Mind2Web +""" + +from __future__ import annotations + +import json +import logging +import os +import re +import shutil +import urllib.request +from pathlib import Path + +TEMPLATE_DIR = Path(__file__).parent / "task-template" +# adapters/online-mind2web/src/online_mind2web/adapter.py -> adapters/online-mind2web +ADAPTER_ROOT = Path(__file__).resolve().parents[2] +JUDGE_BUNDLE = ADAPTER_ROOT / "judge" / "dist" / "judge.js" +logger = logging.getLogger(__name__) + +DATASET_URL = ( + "https://huggingface.co/datasets/osunlp/Online-Mind2Web/" + "resolve/main/Online_Mind2Web.json" +) +# Token env vars, in the precedence the cua loader uses (dataset.ts:46-50). +_TOKEN_ENV_VARS = ("HF_TOKEN", "HUGGINGFACE_TOKEN", "HUGGING_FACE_HUB_TOKEN") +# Local caches checked before fetching (the session ships a pre-fetched copy). +_CACHE_PATHS = ( + Path("/tmp/om2w-real.json"), + Path.home() / ".cache" / "cua-bench" / "online-mind2web" / "Online_Mind2Web.json", +) +_SLUG_RE = re.compile(r"[^a-z0-9-]") + + +class OnlineMind2WebTask: + """One Online-Mind2Web row mapped to the adapter's task shape.""" + + def __init__(self, instruction: str, website: str | None, task_id: str, reference_length: int | None): + self.id = task_id + self.instruction = instruction + self.website = website + self.reference_length = reference_length + + @property + def start_url(self) -> str | None: + """Absolute https:// URL for kernel.json, or None when the row is blank.""" + site = (self.website or "").strip() + if not site: + return None + if "://" not in site: + site = f"https://{site}" + return site + + +def parse_tasks(raw: str) -> list[OnlineMind2WebTask]: + """Map dataset JSON to tasks, skipping rows missing a task_id or instruction. + + Mirrors ``parseOnlineMind2WebTasks`` (cua ``dataset.ts:20-34``) so the task + set matches the cua loader's exactly. + """ + rows = json.loads(raw) + tasks: list[OnlineMind2WebTask] = [] + for row in rows: + instruction = row.get("confirmed_task") or row.get("task") + task_id = row.get("task_id") + if not task_id or not instruction: + continue + ref_len = row.get("reference_length") + tasks.append( + OnlineMind2WebTask( + instruction=instruction, + website=row.get("website"), + task_id=task_id, + reference_length=ref_len if isinstance(ref_len, int) else None, + ) + ) + return tasks + + +def _read_token() -> str | None: + for var in _TOKEN_ENV_VARS: + token = os.environ.get(var) + if token: + return token + return None + + +def load_dataset_raw() -> str: + """Return the dataset JSON text from a local cache or the gated HF download. + + Build-time only: needs ``HF_TOKEN`` on a cache miss. Once tasks are + generated, runners never touch HF again — the instruction and URL are baked + into each task dir. + """ + for cache in _CACHE_PATHS: + if cache.exists(): + logger.info(f"Loading Online-Mind2Web dataset from cache: {cache}") + return cache.read_text() + + token = _read_token() + if not token: + raise RuntimeError( + "Online-Mind2Web is a gated HF dataset. Accept the terms at " + "https://huggingface.co/datasets/osunlp/Online-Mind2Web and set HF_TOKEN." + ) + logger.info("Fetching Online-Mind2Web dataset from HuggingFace...") + req = urllib.request.Request(DATASET_URL, headers={"Authorization": f"Bearer {token}"}) + with urllib.request.urlopen(req) as resp: + if resp.status != 200: + raise RuntimeError( + f"failed to fetch Online-Mind2Web dataset ({resp.status}). " + "Ensure HF_TOKEN has access (gated dataset)." + ) + body = resp.read().decode("utf-8") + + cache = _CACHE_PATHS[-1] + cache.parent.mkdir(parents=True, exist_ok=True) + cache.write_text(body) + return body + + +class OnlineMind2WebAdapter: + """Convert Online-Mind2Web rows into Harbor task directories.""" + + NAME = "online-mind2web" + + @staticmethod + def make_local_task_id(source_id: str) -> str: + """Slug an upstream task_id into a Harbor-safe directory name. + + Upstream ids contain hex hashes and underscores; the registry name + pattern allows only lowercase alphanumerics and hyphens. + """ + normalized = _SLUG_RE.sub("-", source_id.lower().replace("_", "-")).strip("-") + return f"online-mind2web-{normalized}" + + def __init__( + self, + output_dir: Path, + limit: int | None = None, + overwrite: bool = False, + task_ids: list[str] | None = None, + raw_dataset: str | None = None, + **kwargs: object, + ): + self.output_dir = Path(output_dir) + self.limit = limit + self.overwrite = overwrite + self.task_ids = task_ids + self._config = kwargs + + raw = raw_dataset if raw_dataset is not None else load_dataset_raw() + self.tasks = parse_tasks(raw) + + def _prepare_task(self, task: OnlineMind2WebTask, output_dir: Path) -> None: + """Render one task directory from the template.""" + output_dir.mkdir(parents=True, exist_ok=True) + + # environment/kernel.json — per-task browser config (replaces a Dockerfile). + env_dir = output_dir / "environment" + env_dir.mkdir(exist_ok=True) + kernel_cfg: dict[str, object] = { + "stealth": True, + "viewport": {"width": 1280, "height": 1024}, + } + if task.start_url: + kernel_cfg = {"start_url": task.start_url, **kernel_cfg} + (env_dir / "kernel.json").write_text(json.dumps(kernel_cfg, indent=2) + "\n") + + # tests/ — the WebJudge wrapper, the bundled judge, and its grader input. + tests_dir = output_dir / "tests" + tests_dir.mkdir(exist_ok=True) + shutil.copy2(TEMPLATE_DIR / "tests/test.sh", tests_dir / "test.sh") + shutil.copy2(JUDGE_BUNDLE, tests_dir / "judge.js") + task_json = { + "task_id": task.id, + "instruction": task.instruction, + "start_url": task.start_url, + "reference_length": task.reference_length, + } + (tests_dir / "task.json").write_text(json.dumps(task_json, indent=2) + "\n") + + # task.toml — substitute the slugged id and reference length. + task_toml = (TEMPLATE_DIR / "task.toml").read_text() + task_toml = task_toml.replace("{task_id}", self.make_local_task_id(task.id)) + task_toml = task_toml.replace( + "{reference_length}", + str(task.reference_length) if task.reference_length is not None else "", + ) + (output_dir / "task.toml").write_text(task_toml) + + # instruction.md — the agent prompt, with or without a start URL. + prompt_name = "instruction.md" if task.start_url else "instruction.nourl.md" + instruction = (TEMPLATE_DIR / prompt_name).read_text() + instruction = instruction.replace("{start_url}", task.start_url or "") + instruction = instruction.replace("{instruction}", task.instruction) + (output_dir / "instruction.md").write_text(instruction) + + # solution/ — no oracle exists for live-web tasks; ship the no-op stub. + solution_dir = output_dir / "solution" + solution_dir.mkdir(exist_ok=True) + shutil.copy2(TEMPLATE_DIR / "solution/solve.sh", solution_dir / "solve.sh") + + def _select(self) -> list[OnlineMind2WebTask]: + selected = self.tasks + if self.task_ids is not None: + requested = set(self.task_ids) + selected = [ + t + for t in selected + if t.id in requested or self.make_local_task_id(t.id) in requested + ] + if self.limit is not None: + selected = selected[: self.limit] + return selected + + def run(self) -> None: + """Generate the selected task directories under ``output_dir``.""" + if not JUDGE_BUNDLE.exists(): + raise FileNotFoundError( + f"WebJudge bundle not built at {JUDGE_BUNDLE}; run " + "`cd benchmarks/adapters/online-mind2web/judge && npm install && npm run build`" + ) + selected = self._select() + logger.info(f"Generating {len(selected)} tasks...") + self.output_dir.mkdir(parents=True, exist_ok=True) + + for i, task in enumerate(selected): + local_task_id = self.make_local_task_id(task.id) + output_dir = self.output_dir / local_task_id + if output_dir.exists(): + if not self.overwrite: + continue + shutil.rmtree(output_dir) + if (i + 1) % 100 == 0 or i == 0: + logger.info(f"Progress: {i + 1}/{len(selected)} - {local_task_id}") + self._prepare_task(task, output_dir) diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/main.py b/benchmarks/adapters/online-mind2web/src/online_mind2web/main.py new file mode 100644 index 0000000..d7a9466 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/main.py @@ -0,0 +1,81 @@ +"""Generate Harbor tasks from the Online-Mind2Web dataset.""" + +from __future__ import annotations + +import argparse +import logging +import sys +from pathlib import Path + +if __package__ in (None, ""): + sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + from online_mind2web.adapter import OnlineMind2WebAdapter +else: + from .adapter import OnlineMind2WebAdapter + +# adapters/online-mind2web/src/online_mind2web/main.py -> adapters/online-mind2web +ADAPTER_ROOT = Path(__file__).resolve().parents[2] + +logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") +logger = logging.getLogger(__name__) + + +def _default_output_dir() -> Path: + return ADAPTER_ROOT / ".tasks" + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Generate Harbor tasks for the Online-Mind2Web benchmark", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--output-dir", + type=Path, + default=_default_output_dir(), + help="Directory to write generated tasks (defaults to adapters/online-mind2web/.tasks)", + ) + parser.add_argument( + "--limit", + "--num-tasks", + type=int, + dest="limit", + default=None, + help="Generate only the first N tasks", + ) + parser.add_argument( + "--overwrite", + action="store_true", + help="Overwrite existing tasks", + ) + parser.add_argument( + "--task-ids", + nargs="+", + default=None, + help="Only generate these task IDs (upstream or slugged)", + ) + return parser.parse_args() + + +def main() -> None: + args = _parse_args() + + output_dir: Path = args.output_dir + output_dir.mkdir(parents=True, exist_ok=True) + + logger.info("=== Starting Online-Mind2Web Adapter ===") + logger.info(f"Output directory: {output_dir.resolve()}") + + adapter = OnlineMind2WebAdapter( + output_dir=output_dir, + limit=args.limit, + overwrite=args.overwrite, + task_ids=args.task_ids, + ) + logger.info(f"Loaded {len(adapter.tasks)} tasks") + adapter.run() + logger.info(f"Generated tasks in: {output_dir}") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md new file mode 100644 index 0000000..5a948cd --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md @@ -0,0 +1,10 @@ +Go to {start_url} and then complete the following task: + +{instruction} + +When you are done, write a short summary of the outcome (what you accomplished, +and any answer the task asked for) to `/logs/agent/answer.txt`. + +**Important:** +- You should ONLY interact with the live website provided to you AND NEVER ASK FOR HUMAN HELP. +- The website is a real, live site — work with whatever it currently shows. diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md new file mode 100644 index 0000000..06643f4 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md @@ -0,0 +1,10 @@ +Complete the following task on the web: + +{instruction} + +When you are done, write a short summary of the outcome (what you accomplished, +and any answer the task asked for) to `/logs/agent/answer.txt`. + +**Important:** +- You should ONLY interact with live websites AND NEVER ASK FOR HUMAN HELP. +- The websites are real, live sites — work with whatever they currently show. diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/solution/solve.sh b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/solution/solve.sh new file mode 100644 index 0000000..7917959 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/solution/solve.sh @@ -0,0 +1,8 @@ +#!/bin/bash +set -e + +# No oracle exists for live-web Online-Mind2Web (no gold trajectory or expected +# answer). This stub only satisfies the harness shape for the `oracle` agent; +# WebJudge will score it a failure, which is expected. +mkdir -p /logs/agent +echo "no oracle available for live-web Online-Mind2Web" > /logs/agent/answer.txt diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml new file mode 100644 index 0000000..4983e07 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml @@ -0,0 +1,28 @@ +schema_version = "1.0" + +[task] +name = "osunlp/{task_id}" +authors = [{ name = "OSU NLP Group" }] +keywords = ["online-mind2web", "web-agent", "live-web"] + +[metadata] +difficulty = "hard" +category = "web-navigation" +source = "osunlp/Online-Mind2Web" +reference_length = "{reference_length}" + +[agent] +timeout_sec = 1800.0 + +[verifier] +timeout_sec = 900.0 + +[verifier.env] +# WebJudge backbone. The judge bin parses the provider from JUDGE_MODEL and +# reads the matching provider key; anthropic is the wired provider. +ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" +JUDGE_MODEL = "anthropic:claude-sonnet-4-6" +SCORE_THRESHOLD = "3" + +[environment] +build_timeout_sec = 300.0 diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh new file mode 100644 index 0000000..82b40ad --- /dev/null +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -uo pipefail + +# WebJudge verifier for Online-Mind2Web. Runs the self-contained judge bundle +# (uploaded alongside this script to /tests) inside the Kernel VM, grading the +# agent's trajectory under /logs/agent. The bundle writes the reward itself and +# falls back to 0 on any error; this wrapper writes 0 too if node never runs so +# the reward file is never left empty. +mkdir -p /logs/verifier + +node /tests/judge.js \ + --task /tests/task.json \ + --run /logs/agent/run.jsonl \ + --answer /logs/agent/answer.txt \ + --shots /logs/agent \ + --judge-model "${JUDGE_MODEL:-anthropic:claude-sonnet-4-6}" \ + --score-threshold "${SCORE_THRESHOLD:-3}" \ + --reward-out /logs/verifier/reward.txt \ + --details-out /logs/verifier/grading_details.json + +if [ ! -s /logs/verifier/reward.txt ]; then + echo 0 > /logs/verifier/reward.txt +fi diff --git a/benchmarks/adapters/online-mind2web/tests/test_adapter.py b/benchmarks/adapters/online-mind2web/tests/test_adapter.py new file mode 100644 index 0000000..8ffd68c --- /dev/null +++ b/benchmarks/adapters/online-mind2web/tests/test_adapter.py @@ -0,0 +1,145 @@ +"""Mocked unit tests for the Online-Mind2Web adapter (no live HF / no network).""" + +from __future__ import annotations + +import json +import tomllib +from pathlib import Path + +import pytest + +from online_mind2web import adapter as adapter_mod +from online_mind2web.adapter import OnlineMind2WebAdapter, parse_tasks + +SAMPLE = json.dumps( + [ + { + "task_id": "abc123_110325", + "confirmed_task": "Find the closest store to 90028.", + "website": "https://www.traderjoes.com/", + "reference_length": 6, + }, + # bare host -> normalized to https:// + { + "task_id": "bare_host_1", + "confirmed_task": "Compare plans.", + "website": "apple.com", + "reference_length": 4, + }, + # blank website -> start_url omitted, instruction-only prompt + { + "task_id": "no_url_1", + "confirmed_task": "Do a thing.", + "website": "", + }, + # falls back to `task` when confirmed_task is absent + { + "task_id": "fallback_1", + "task": "Fallback instruction.", + "website": "https://example.com", + }, + # malformed: missing task_id -> skipped + {"confirmed_task": "no id", "website": "https://x.com"}, + # malformed: missing instruction -> skipped + {"task_id": "no_instruction", "website": "https://y.com"}, + ] +) + + +@pytest.fixture(autouse=True) +def _fake_judge_bundle(monkeypatch, tmp_path): + """Stand in a dummy judge bundle so _prepare_task's copy + run()'s check pass.""" + bundle = tmp_path / "judge.js" + bundle.write_text("// dummy bundle\n") + monkeypatch.setattr(adapter_mod, "JUDGE_BUNDLE", bundle) + + +def _generate(tmp_path: Path, **kwargs) -> Path: + out = tmp_path / "tasks" + OnlineMind2WebAdapter(output_dir=out, raw_dataset=SAMPLE, **kwargs).run() + return out + + +def test_parse_skips_malformed_rows(): + tasks = parse_tasks(SAMPLE) + ids = [t.id for t in tasks] + assert ids == ["abc123_110325", "bare_host_1", "no_url_1", "fallback_1"] + + +def test_fallback_instruction_uses_task_field(): + tasks = {t.id: t for t in parse_tasks(SAMPLE)} + assert tasks["fallback_1"].instruction == "Fallback instruction." + + +def test_start_url_normalization(): + tasks = {t.id: t for t in parse_tasks(SAMPLE)} + assert tasks["abc123_110325"].start_url == "https://www.traderjoes.com/" + assert tasks["bare_host_1"].start_url == "https://apple.com" + assert tasks["no_url_1"].start_url is None + + +def test_slug_strips_underscores_and_hex(): + assert ( + OnlineMind2WebAdapter.make_local_task_id("ABC_123def") + == "online-mind2web-abc-123def" + ) + + +def test_generates_full_task_dir(tmp_path): + out = _generate(tmp_path) + task_dir = out / "online-mind2web-abc123-110325" + assert task_dir.is_dir() + + for rel in ("instruction.md", "task.toml", "environment/kernel.json", + "tests/test.sh", "tests/task.json", "tests/judge.js", "solution/solve.sh"): + assert (task_dir / rel).is_file(), rel + + kernel = json.loads((task_dir / "environment/kernel.json").read_text()) + assert kernel == { + "start_url": "https://www.traderjoes.com/", + "stealth": True, + "viewport": {"width": 1280, "height": 1024}, + } + + task_json = json.loads((task_dir / "tests/task.json").read_text()) + assert task_json["task_id"] == "abc123_110325" + assert task_json["start_url"] == "https://www.traderjoes.com/" + assert task_json["reference_length"] == 6 + + toml = tomllib.loads((task_dir / "task.toml").read_text()) + assert toml["task"]["name"] == "osunlp/online-mind2web-abc123-110325" + assert toml["metadata"]["reference_length"] == "6" + assert toml["verifier"]["env"]["JUDGE_MODEL"].startswith("anthropic:") + + instruction = (task_dir / "instruction.md").read_text() + assert "https://www.traderjoes.com/" in instruction + assert "Find the closest store to 90028." in instruction + assert "/logs/agent/answer.txt" in instruction + + +def test_blank_website_omits_start_url_and_uses_nourl_prompt(tmp_path): + out = _generate(tmp_path) + task_dir = out / "online-mind2web-no-url-1" + kernel = json.loads((task_dir / "environment/kernel.json").read_text()) + assert "start_url" not in kernel + instruction = (task_dir / "instruction.md").read_text() + assert "Go to" not in instruction + assert "Do a thing." in instruction + + +def test_limit_and_overwrite(tmp_path): + out = _generate(tmp_path, limit=2) + dirs = sorted(p.name for p in out.iterdir() if p.is_dir()) + assert dirs == ["online-mind2web-abc123-110325", "online-mind2web-bare-host-1"] + + +def test_task_ids_selects_by_upstream_or_slug(tmp_path): + out = _generate(tmp_path, task_ids=["no_url_1", "online-mind2web-bare-host-1"]) + dirs = sorted(p.name for p in out.iterdir() if p.is_dir()) + assert dirs == ["online-mind2web-bare-host-1", "online-mind2web-no-url-1"] + + +def test_run_raises_when_bundle_missing(tmp_path, monkeypatch): + monkeypatch.setattr(adapter_mod, "JUDGE_BUNDLE", tmp_path / "absent.js") + with pytest.raises(FileNotFoundError, match="WebJudge bundle not built"): + OnlineMind2WebAdapter(output_dir=tmp_path / "o", raw_dataset=SAMPLE).run() From fca5cb24d585c739e278517b7c2be0b91db4cddd Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 12:44:09 +0000 Subject: [PATCH 2/9] online-mind2web: opus judge + live-smoke fixes Finalize the adapter against an opus agent + opus WebJudge after a 20-task live smoke (16/20, mean 0.800; pools worked, no 403). Fixes the smoke surfaced: - WebJudge: retry the Anthropic call without `temperature` on the 400 newer models (e.g. opus-4-8) return for the deprecated param, instead of fail-closing every grade to reward 0. Keep temperature=0 where supported. - Instruction: tell the browser-only agent to state its answer in its final message (captured into answer.txt) rather than to write a file it has no tools for. - Default JUDGE_MODEL to anthropic:claude-opus-4-8 (task.toml, test.sh, judge). - Gitignore .validate/, _smoke_logs/, and the standalone uv.lock. Adds a mocked anthropicJudgeModel test for the temperature retry. SMOKE.md records the run, the two fixes, and the failure taxonomy. Co-Authored-By: Claude Opus 4.7 --- .../adapters/online-mind2web/.gitignore | 5 +- benchmarks/adapters/online-mind2web/README.md | 5 +- benchmarks/adapters/online-mind2web/SMOKE.md | 198 ++++++++++++------ .../online-mind2web/judge/src/judge.ts | 2 +- .../online-mind2web/judge/src/model.ts | 47 +++-- .../judge/test/artifacts.test.ts | 52 ++++- .../task-template/instruction.md | 6 +- .../task-template/instruction.nourl.md | 6 +- .../online_mind2web/task-template/task.toml | 2 +- .../task-template/tests/test.sh | 2 +- .../online-mind2web/tests/test_adapter.py | 2 +- 11 files changed, 233 insertions(+), 94 deletions(-) diff --git a/benchmarks/adapters/online-mind2web/.gitignore b/benchmarks/adapters/online-mind2web/.gitignore index 98d3e01..92220b6 100644 --- a/benchmarks/adapters/online-mind2web/.gitignore +++ b/benchmarks/adapters/online-mind2web/.gitignore @@ -1,5 +1,7 @@ -# Generated task dirs + caches (never committed) +# Generated task dirs, smoke runs, caches (never committed) .tasks/ +.validate/ +_smoke_logs/ jobs/ trials/ dataset/ @@ -11,6 +13,7 @@ __pycache__/ .venv/ .pytest_cache/ .ruff_cache/ +uv.lock # Node judge bin build artifacts judge/dist/ diff --git a/benchmarks/adapters/online-mind2web/README.md b/benchmarks/adapters/online-mind2web/README.md index 072e3e9..720b293 100644 --- a/benchmarks/adapters/online-mind2web/README.md +++ b/benchmarks/adapters/online-mind2web/README.md @@ -53,5 +53,8 @@ cd benchmarks && uv sync && (cd node && npm install && npm run build) uv run harbor run -p adapters/online-mind2web/.tasks -e kernel \ --environment-kwarg pool_size=8 \ --agent-import-path cua_harbor:CuaHarborAgent \ - -m anthropic/claude-sonnet-4-6 --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY -n 6 + -m anthropic/claude-opus-4-8 --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY -n 6 ``` + +If browser pools 403 on your account (quota/plan), drop `--environment-kwarg +pool_size=8` to fall back to per-task browser creation. diff --git a/benchmarks/adapters/online-mind2web/SMOKE.md b/benchmarks/adapters/online-mind2web/SMOKE.md index 8717177..6806aa9 100644 --- a/benchmarks/adapters/online-mind2web/SMOKE.md +++ b/benchmarks/adapters/online-mind2web/SMOKE.md @@ -1,67 +1,131 @@ -# online-mind2web — smoke notes - -The live 20-task Harbor smoke was **not run** in this session (scope: implement -+ unit-test the adapter only; the parent runs the live smoke after review). -What was validated, and the learnings that shaped the implementation, are below. - -## Validated without the live harbor run - -- **Judge against the live Anthropic API.** The bundled `judge.js` ran - standalone against `anthropic:claude-sonnet-4-6`: it made the WebJudge - KEY_POINTS + FINAL model calls, parsed the verdict, and wrote - `reward.txt` + `grading_details.json`. The `model.ts` fetch client, the - recovered three-stage pipeline, the parsers, and the reward writer all work - end-to-end against the real provider. -- **Kernel VM runtime probe.** A throwaway browser session confirmed the VM - ships `node v22.23.1` (with global `fetch`), `python3 3.10`, and `curl`, and - that `/logs/agent` + `/logs/verifier` are writable and ESM bundles run. This - is the fact the verifier design hinges on (see learnings). -- **Generation end-to-end.** `python -m online_mind2web.main --limit 3` against - the cached dataset produced full task dirs (`instruction.md`, `task.toml`, - `environment/kernel.json`, `tests/{test.sh,judge.js,task.json}`, - `solution/solve.sh`) with correct `start_url` normalization and slugged ids. -- **Unit tests / lint / typecheck.** `ruff check` clean; 9 mocked Python adapter - tests pass (no network — dataset injected); judge `tsc --noEmit` clean; 13 - vitest tests pass (parsers, `gradeWithWebJudge` with a scripted judge, and - `loadTrajectory`/`loadTask` against a real on-disk `/logs/agent` layout). - -## Learnings that changed the design vs `map-online-mind2web.md` - -The design doc predates the finalized shared core (PR #40) and assumed two -artifacts that do not exist; the implementation reconciles to what the shared -core actually emits. - -1. **No `trajectory.bench.json`, no `@onkernel/cua-bench` `grade.js`.** The - shared `node/src/sink.ts` writes `answer.txt`, `run.jsonl`, and spilled - `shots/shot-.` (1-indexed) — not a purpose-built grader index, and - the entrypoint package is `cua-bench-task`, not `@onkernel/cua-bench`. The - verifier therefore reconstructs the WebJudge `Trajectory` directly from - `run.jsonl` (pairing each `tool_result`'s spilled shot with the originating - `assistant` tool call as the "action") + `answer.txt` (`judge/src/artifacts.ts`). -2. **The Kernel VM does have `node`.** The benchmarks README says the base VM - "ships no Node", which is true for the cua *agent* (its npm package isn't - installed in-VM, so the agent entrypoint runs on the host). But a raw `node` - binary is present, so the verifier's `test.sh` runs the WebJudge in-VM. The - judge is bundled to a single self-contained ESM file (tsdown, no externals, - ~16 kB) and copied into each task's `tests/` so it travels with the uploaded - tests dir and needs no install at verify time. -3. **Anthropic judge, not OpenAI.** No `OPENAI_API_KEY` this session, so the - WebJudge backbone is `anthropic:claude-sonnet-4-6` via a minimal Anthropic - Messages client over global `fetch` (`judge/src/model.ts`) instead of the - recovered cua-ai `piJudgeModel`. `JUDGE_MODEL` / `SCORE_THRESHOLD` stay - configurable in `[verifier.env]`. The recovered `webjudge.ts` / `prompts.ts` - are otherwise placed verbatim (they are line-checked against upstream and - carry the parser-coupled literal markers). -4. **Answer-file path = `/logs/agent/answer.txt`.** Matches the shared-core - contract (`cua_harbor.constants.ANSWER_FILE`) and the example task's - `test.sh`, resolving the open item in the design doc. - -## What the live smoke should still surface (to capture next round) - -- Pass rate / per-task reward across ~20 tasks with `pool_size=8` (fall back to - `pool_size=1` if pools 403 on this account; note it, don't fail the smoke). -- Failure taxonomy: env-vs-task, live-site drift / anti-bot walls (stealth is on - in every `kernel.json`), judge disagreement, adapter bugs. -- Whether `claude-sonnet-4-6` as the judge backbone tracks the published - WebJudge(o4-mini) success rate within a tolerance band (live-site drift makes - exact parity impossible; pin judge + viewport + threshold and record the date). +# online-mind2web — live smoke + +Live Harbor smoke of the full pipeline (Kernel browser env + cua agent + WebJudge +verifier) on 20 generated Online-Mind2Web tasks. Date: 2026-06-27. NOT a +definitive benchmark number — the goal is to exercise the real pipeline, learn +failure modes, and fix adapter bugs the smoke surfaces. + +## Configuration + +- **Agent:** cua (`--agent-import-path cua_harbor:CuaHarborAgent`), model + `anthropic/claude-opus-4-8`. +- **Judge:** WebJudge with an Anthropic multimodal backbone, + `anthropic:claude-opus-4-8` (`JUDGE_MODEL` in `[verifier.env]`, + `SCORE_THRESHOLD=3`). Three-stage pipeline (key-point extraction → + per-screenshot 1-5 scoring → final verdict), recovered verbatim from upstream. +- **Tasks:** 20 task dirs generated from the cached 300-task dataset + (`/tmp/om2w-real.json`) via `online_mind2web.main --limit 20`. Live consumer + sites: airlines (Qatar, United), retail (Amazon, Uniqlo, Speedo), cars + (KBB, Carvana, CarMax), media (IMDb, IGN), transit (Amtrak, MTA), etc. +- **Pools:** `--environment-kwarg pool_size=5` — **pools worked, no 403** on this + account. (Fallback to `pool_size=1` was not needed.) +- **Run:** `-n 4`, 1800s/task agent cap, 900s/task verifier cap. + +``` +uv run harbor run -y -p adapters/online-mind2web/.tasks -e kernel \ + --environment-kwarg pool_size=5 \ + --agent-import-path cua_harbor:CuaHarborAgent \ + -m anthropic/claude-opus-4-8 --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY -n 4 +``` + +## Result + +**16 / 20 pass — mean reward 0.800.** 1 exception (`AgentTimeoutError`). Runtime +48m53s. Agent spend (opus, from trajectory metrics) ~$11.6 over 20 live +sessions; the opus WebJudge is billed separately on top — a sonnet judge would +cut the grading bill materially with little expected accuracy loss. Rewards: +16×`1.0`, 4×`0.0`. + +| Outcome | Count | +|---|---| +| SUCCESS (reward 1.0) | 16 | +| NOT SUCCESS (reward 0.0) | 4 | +| of which agent exceptions | 1 (`AgentTimeoutError`) | + +Every verifier ran clean (no judge errors in any `test-stdout.txt`) — the +opus-judge `temperature` fix below held across all 20 trials, and the pipeline +(browser provision -> agent drive -> answer+shots land -> WebJudge score) was +green end-to-end on every task. The 4 failures are agent/task/judge effects, not +adapter bugs. + +## Validation gate (2 tasks, run first) + +Before the 20-task run, 2 tasks (IMDb crazy-credits, Trader Joe's store locator) +were run end-to-end to confirm the pipeline was green. The first validation pass +**surfaced a real adapter bug** (see below): both tasks scored 0.0 despite the +agent clearly completing them. After the fix, the same 2 tasks scored **2/2 +(1.0)**. Then the full 20-task run was launched. + +## Adapter bugs surfaced and FIXED by the smoke + +1. **WebJudge crashed on the opus judge: `temperature` is deprecated.** The + Anthropic judge client (`judge/src/model.ts`) sent `temperature: 0` on every + call (WebJudge wants deterministic scoring). `claude-opus-4-8` rejects the + parameter outright with HTTP 400 (`"temperature is deprecated for this + model"`). The judge's fail-closed `catch` then wrote reward `0` for every + task — so the whole run scored 0.0 even though `answer.txt`, `run.jsonl`, and + the per-step screenshots all landed correctly and the agent had succeeded. + **Fix:** on a 400 whose body mentions `temperature`, retry the request once + without the field; keep `temperature: 0` for models that accept it. Covered + by a new mocked `anthropicJudgeModel` test (fetch stubbed). + +2. **Instruction told a browser-only agent to write a file.** The template ended + with "write a short summary … to `/logs/agent/answer.txt`". The cua agent has + only computer-use (browser) tools, so it burned several turns trying to + "navigate to a file URL" to write the file, then gave up. The answer landed + anyway — the Node entrypoint's `extractFinalAnswer` captures the last + assistant message into `answer.txt` regardless — so the misdirection only + wasted steps. **Fix:** the instruction now tells the agent to state its + answer directly in its final message (which is what is recorded) and not to + attempt file I/O. Applied to both `instruction.md` and `instruction.nourl.md`. + +## Failure taxonomy + +All 4 zero-reward trials, classified (per-task reward + judge reasoning from +`grading_details.json`): + +1. **env / budget — agent timeout (1 task).** IGN "editor's choice review with a + score of 10 in the boardgame category." The agent ran to the 1800s cap + (`AgentTimeoutError`) after 63 screenshots; the verifier still graded the + partial trajectory and scored NOT SUCCESS. This is a budget/latency effect on + a deep-navigation site, not an adapter fault. A higher `[agent].timeout_sec` + (or a faster agent model) would likely recover it. +2. **agent stall — empty final answer (1 task).** Speedo "women's black + one-piece swimsuit, size large, highest rating." 47 screenshots landed but + `answer.txt` was 0 bytes — the agent's last turn was a tool call, not a text + message, so `extractFinalAnswer` had nothing to capture. The judge fail-closes + to 0 on screenshots-with-no-answer. Inherent agent behavior; the instruction + already asks for a final summary, and the screenshots/`run.jsonl` were still + captured correctly. +3. **judge strict-grading — filter not visibly applied (1 task).** Micro Center + "most helpful reviews of the PS5 Digital Edition." The agent's answer was + reasonable (the live product had only a single ratings-only review, so there + was nothing to sort), but WebJudge criteria 1-3 require the *sort/filter to be + visibly applied*; the agent filtered by 5-star rating instead of a "Most + Helpful" sort, so the judge scored NOT SUCCESS. This is the WebJudge's known + strict posture interacting with live-site reality, not an agent capability gap. +4. **genuine task failure — hard multi-constraint (1 task).** Carvana "cheapest + used Honda Civic meeting all of: <6 constraints>." Adding the final + "Blindspot Sensors" filter yielded 0 matches, so the agent dropped it and + picked a car missing that constraint. WebJudge criterion 3 (all requirements + must be applied via the filter) correctly scores this 0 — a real failure of + the agent on an unsatisfiable-as-posed live query. + +Not observed: no anti-bot / CAPTCHA hard-block surfaced in this sample (stealth +held on every site), and **no adapter bugs and no judge disagreement-by-crash** — +the one judge "disagreement" (PS5) is the judge working as specified. + +## What held up (no change needed) + +- **Pipeline wiring is correct.** The shared-core contract (`answer.txt`, + `run.jsonl`, `shots/*.png` under `/logs/agent`) and the verifier's reads of + exactly those paths agree end-to-end; the judge reconstructs the WebJudge + trajectory (pairing each spilled screenshot with the tool call that produced + it) and scores it. +- **Self-contained in-VM judge.** The bundled `judge.js` (~16 kB ESM, no + externals) runs under the VM's `node` + global `fetch` with no install at + verify time; the reward file is never left empty (fail-closed to 0). +- **`start_url` + stealth + pinned viewport.** Every `kernel.json` lands the + browser on the right page with `stealth: true` and a fixed 1280×1024 viewport + so the screenshots the judge sees are reproducible. +- **Browser pools.** `pool_size=5` provisioned without 403 on this account. diff --git a/benchmarks/adapters/online-mind2web/judge/src/judge.ts b/benchmarks/adapters/online-mind2web/judge/src/judge.ts index 293d02c..348ae95 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/judge.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/judge.ts @@ -46,7 +46,7 @@ function parseArgs(argv: string[]): Args { run: require("run"), answer: require("answer"), shots: flags.get("shots"), - judgeModel: flags.get("judge-model") ?? "anthropic:claude-sonnet-4-6", + judgeModel: flags.get("judge-model") ?? "anthropic:claude-opus-4-8", scoreThreshold: Number(flags.get("score-threshold") ?? "3"), rewardOut: require("reward-out"), detailsOut: flags.get("details-out"), diff --git a/benchmarks/adapters/online-mind2web/judge/src/model.ts b/benchmarks/adapters/online-mind2web/judge/src/model.ts index 19a631d..0769b4f 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/model.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/model.ts @@ -45,23 +45,40 @@ export function anthropicJudgeModel(ref: string): JudgeModel { if (!apiKey) { throw new Error("ANTHROPIC_API_KEY is required for the WebJudge model"); } + const headers = { + "content-type": "application/json", + "x-api-key": apiKey, + "anthropic-version": ANTHROPIC_VERSION, + }; + + async function callOnce( + systemPrompt: string, + content: JudgeContent, + withTemperature: boolean, + ): Promise { + const body: Record = { + model: name, + max_tokens: 1024, + system: systemPrompt, + messages: [{ role: "user", content: toAnthropicContent(content) }], + }; + // WebJudge wants deterministic scoring, but newer models (e.g. opus-4-8) + // reject `temperature` outright; on that 400 we retry without it. + if (withTemperature) body.temperature = 0; + return fetch(ANTHROPIC_URL, { method: "POST", headers, body: JSON.stringify(body) }); + } + return { async complete(systemPrompt, content) { - const res = await fetch(ANTHROPIC_URL, { - method: "POST", - headers: { - "content-type": "application/json", - "x-api-key": apiKey, - "anthropic-version": ANTHROPIC_VERSION, - }, - body: JSON.stringify({ - model: name, - max_tokens: 1024, - temperature: 0, - system: systemPrompt, - messages: [{ role: "user", content: toAnthropicContent(content) }], - }), - }); + let res = await callOnce(systemPrompt, content, true); + if (res.status === 400) { + const body = await res.text(); + if (body.includes("temperature")) { + res = await callOnce(systemPrompt, content, false); + } else { + throw new Error(`Anthropic API error 400: ${body}`); + } + } if (!res.ok) { const body = await res.text(); throw new Error(`Anthropic API error ${res.status}: ${body}`); diff --git a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts index dc86474..3b2ae0e 100644 --- a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts +++ b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts @@ -2,9 +2,9 @@ import { mkdtempSync, writeFileSync } from "node:fs"; import { mkdirSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { loadTask, loadTrajectory } from "../src/artifacts.ts"; -import { parseModelRef } from "../src/model.ts"; +import { anthropicJudgeModel, parseModelRef } from "../src/model.ts"; // 1x1 transparent PNG, the same fixture the shared-core sink test uses. const PNG_B64 = @@ -130,3 +130,51 @@ describe("parseModelRef", () => { }); }); }); + +describe("anthropicJudgeModel", () => { + afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); + }); + + function okResponse(text: string): Response { + return new Response(JSON.stringify({ content: [{ type: "text", text }] }), { status: 200 }); + } + + it("retries without temperature when the model rejects it (400)", async () => { + vi.stubEnv("ANTHROPIC_API_KEY", "k"); + const bodies: Array> = []; + const fetchMock = vi.fn(async (_url: string, init: RequestInit) => { + bodies.push(JSON.parse(init.body as string)); + if (bodies.length === 1) { + return new Response( + JSON.stringify({ error: { message: "`temperature` is deprecated for this model." } }), + { status: 400 }, + ); + } + return okResponse("verdict"); + }); + vi.stubGlobal("fetch", fetchMock); + + const out = await anthropicJudgeModel("anthropic:claude-opus-4-8").complete("sys", [ + { type: "text", text: "hi" }, + ]); + + expect(out).toBe("verdict"); + expect(fetchMock).toHaveBeenCalledTimes(2); + expect(bodies[0].temperature).toBe(0); + expect(bodies[1]).not.toHaveProperty("temperature"); + }); + + it("throws on a 400 unrelated to temperature", async () => { + vi.stubEnv("ANTHROPIC_API_KEY", "k"); + vi.stubGlobal( + "fetch", + vi.fn(async () => new Response(JSON.stringify({ error: { message: "bad request" } }), { status: 400 })), + ); + + await expect( + anthropicJudgeModel("anthropic:claude-opus-4-8").complete("sys", [{ type: "text", text: "hi" }]), + ).rejects.toThrow("Anthropic API error 400"); + }); +}); diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md index 5a948cd..5a5006a 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.md @@ -2,8 +2,10 @@ Go to {start_url} and then complete the following task: {instruction} -When you are done, write a short summary of the outcome (what you accomplished, -and any answer the task asked for) to `/logs/agent/answer.txt`. +When you are done, end with a short summary of the outcome: what you +accomplished, and any answer the task asked for. State it directly in your final +message — that final message is recorded as your answer (you do not have file +tools, so do not try to write a file). **Important:** - You should ONLY interact with the live website provided to you AND NEVER ASK FOR HUMAN HELP. diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md index 06643f4..a0cbd90 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/instruction.nourl.md @@ -2,8 +2,10 @@ Complete the following task on the web: {instruction} -When you are done, write a short summary of the outcome (what you accomplished, -and any answer the task asked for) to `/logs/agent/answer.txt`. +When you are done, end with a short summary of the outcome: what you +accomplished, and any answer the task asked for. State it directly in your final +message — that final message is recorded as your answer (you do not have file +tools, so do not try to write a file). **Important:** - You should ONLY interact with live websites AND NEVER ASK FOR HUMAN HELP. diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml index 4983e07..99f79a7 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml @@ -21,7 +21,7 @@ timeout_sec = 900.0 # WebJudge backbone. The judge bin parses the provider from JUDGE_MODEL and # reads the matching provider key; anthropic is the wired provider. ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" -JUDGE_MODEL = "anthropic:claude-sonnet-4-6" +JUDGE_MODEL = "anthropic:claude-opus-4-8" SCORE_THRESHOLD = "3" [environment] diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh index 82b40ad..2558091 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh @@ -13,7 +13,7 @@ node /tests/judge.js \ --run /logs/agent/run.jsonl \ --answer /logs/agent/answer.txt \ --shots /logs/agent \ - --judge-model "${JUDGE_MODEL:-anthropic:claude-sonnet-4-6}" \ + --judge-model "${JUDGE_MODEL:-anthropic:claude-opus-4-8}" \ --score-threshold "${SCORE_THRESHOLD:-3}" \ --reward-out /logs/verifier/reward.txt \ --details-out /logs/verifier/grading_details.json diff --git a/benchmarks/adapters/online-mind2web/tests/test_adapter.py b/benchmarks/adapters/online-mind2web/tests/test_adapter.py index 8ffd68c..771b7ce 100644 --- a/benchmarks/adapters/online-mind2web/tests/test_adapter.py +++ b/benchmarks/adapters/online-mind2web/tests/test_adapter.py @@ -114,7 +114,7 @@ def test_generates_full_task_dir(tmp_path): instruction = (task_dir / "instruction.md").read_text() assert "https://www.traderjoes.com/" in instruction assert "Find the closest store to 90028." in instruction - assert "/logs/agent/answer.txt" in instruction + assert "final message" in instruction def test_blank_website_omits_start_url_and_uses_nourl_prompt(tmp_path): From dcf55d6d65aac170735c353c92f77af4138b97fc Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 15:47:48 +0000 Subject: [PATCH 3/9] online-mind2web: carry dataset level into [metadata].difficulty Thread each row's level (easy/medium/hard) into task.toml difficulty and tests/task.json instead of hardcoding "hard", which mislabeled most tasks. Default to hard when absent. Document the 1024 max_tokens headroom and add PARITY.md recording the parity pass (applied vs skipped). Co-Authored-By: Claude Opus 4.7 --- benchmarks/adapters/online-mind2web/PARITY.md | 273 ++++++++++++++++++ .../online-mind2web/judge/src/artifacts.ts | 1 + .../online-mind2web/judge/src/model.ts | 3 + .../src/online_mind2web/adapter.py | 17 +- .../online_mind2web/task-template/task.toml | 2 +- .../online-mind2web/tests/test_adapter.py | 9 + 6 files changed, 302 insertions(+), 3 deletions(-) create mode 100644 benchmarks/adapters/online-mind2web/PARITY.md diff --git a/benchmarks/adapters/online-mind2web/PARITY.md b/benchmarks/adapters/online-mind2web/PARITY.md new file mode 100644 index 0000000..7ad7b78 --- /dev/null +++ b/benchmarks/adapters/online-mind2web/PARITY.md @@ -0,0 +1,273 @@ +# online-mind2web — parity with the canonical benchmark + +Line-by-line comparison of this adapter against the official upstream +**OSU-NLP-Group/Online-Mind2Web** (commit on `main`, fetched 2026-06-27 via +`gh api`). The focus is the WebJudge auto-grader +(`src/methods/webjudge_online_mind2web.py`), its driver (`src/run.py`), the +shared `encode_image`/`extract_predication` helpers (`src/utils.py`), and the +dataset/task fields (`osunlp/Online-Mind2Web` rows + `data/schema_v2`). + +Our WebJudge is a TypeScript re-port bundled as a self-contained `judge.js` +(`judge/src/{prompts,webjudge,artifacts,model,judge}.ts`); the Python adapter +(`src/online_mind2web/adapter.py`) does dataset→task-dir generation. + +**Verdict: the WebJudge core is a faithful port.** The three system prompts, the +user-text templates, key-point extraction, per-image 1–5 scoring, the +`score_threshold` keep-filter, `MAX_IMAGE = 50`, and the `Status:` verdict parse +are all reproduced verbatim or behaviorally-identically. The differences that +matter are: (1) **the screenshots are sent as PNG, not JPEG** — upstream +re-encodes every image to JPEG before the judge sees it; (2) **the judge model +is Anthropic opus, not OpenAI o4-mini** — a deliberate Kernel choice that moves +us off the published parity baseline; (3) `max_tokens` is 1024 vs upstream's 512. +None of these change the *pipeline*; (1) and (2) change the *inputs the judge +scores*, so they bear on numeric parity. Details below. + +### Parity pass — applied vs skipped (2026-06-27) + +A follow-up parity pass reviewed the three "minor" items in §2. Outcome: + +- **Applied — §2.5 carry `level` → `[metadata].difficulty`.** The hardcoded + `difficulty = "hard"` mislabeled 221/300 live rows (live `level` split: + medium 141, easy 80, hard 79). The adapter now threads the row's `level` into + `[metadata].difficulty` and `tests/task.json` (`level`), defaulting to `"hard"` + when absent. Metadata only — does not touch grading or selection. +- **Skipped — §2.1 JPEG re-encode.** Requires an image codec (RGBA→RGB + + JPEG) inside the judge, which conflicts with the deliberate self-contained, + dependency-free in-VM bundle, and buys no decimal parity while the judge model + is opus rather than the o4-mini/gpt-4o the published numbers are calibrated to + (§2.2). Not warranted now; revisit only alongside an OpenAI judge branch. +- **Skipped (documented, no behavior change) — §2.3 `max_tokens`.** Left at + 1024 with a code comment explaining the headroom: lowering to upstream's 512 + risks truncating the verbose opus judge's per-image reasoning before the + `Score`/`Status:` line the parsers key on — a regression against the judge we + actually ship, with no parity payoff (opus ≠ o4-mini). + +--- + +## 1. WebJudge grading pipeline + +Upstream `WebJudge_Online_Mind2Web_eval` (`webjudge.py:88`) + +`run.py:auto_eval` vs our `gradeWithWebJudge` (`judge/src/webjudge.ts:8`) + +`judge.ts:main`. + +| Aspect | Upstream | Ours | Match? | +|---|---|---|---| +| Stage 1 — key-point system prompt | `webjudge.py:8-20` | `KEY_POINTS_SYSTEM`, `prompts.ts:11` | verbatim ✓ | +| Stage 1 — user text | `"Task: {task}"` (`webjudge.py:21`) | `keyPointsUserText` → `Task: ${task}` (`prompts.ts:76`) | verbatim ✓ | +| Stage 1 — post-process (`\n\n`→`\n`, split on `**Key Points**:` else `Key Points:`, lstrip lines) | `webjudge.py:126-133` | `extractKeyPoints`, `prompts.ts:110-119` | behaviorally identical ✓ | +| Stage 2 — image system prompt | `webjudge.py:36-60` | `JUDGE_IMAGE_SYSTEM`, `prompts.ts:25` | verbatim ✓ | +| Stage 2 — image user text | `webjudge.py:64-68` | `judgeImageUserText`, `prompts.ts:80` | verbatim ✓ | +| Stage 2 — score parse `split("Score")[1]` + `re.findall("[1-5]")[0]` | `webjudge.py:144-146` | `parseImageScore`, `prompts.ts:121-135` (`split("Score")[1]` + `/[1-5]/`) | behaviorally identical ✓ | +| Stage 2 — thought parse `split("**Reasoning**:")[-1].strip().lstrip("\n").split("\n\n")[0].replace("\n"," ")` | `webjudge.py:145` | same chain (`prompts.ts:126-130`) | behaviorally identical ✓ | +| Stage 2 — score everything, no cap on number of scoring calls | all of `images_path` (`webjudge.py:135-136`) | all of `trajectory.steps` with a screenshot (`webjudge.ts:19-33`) | ✓ | +| Keep-filter | `int(score) >= score_threshold` (`webjudge.py:153`) | `r.score >= scoreThreshold` (`webjudge.ts:35`) | ✓ | +| `MAX_IMAGE` cap on kept images **and** kept thoughts | `= 50`; `whole_content_img[:50]`, `whole_thoughts[:50]` (`webjudge.py:5,164-165`) | `MAX_IMAGE = 50`; `kept.slice(0,50)`, `keptThoughts.slice(0,50)` (`prompts.ts:9`, `webjudge.ts:35-36`) | ✓ | +| Empty-kept branch drops the "snapshots" section of the final prompt | `if len(whole_content_img)==0: prompt = ` (`webjudge.py:166-172`) | `hasImages: kept.length > 0` → `finalUserText` returns head only (`prompts.ts:102`, `webjudge.ts:47`) | ✓ | +| Stage 3 — final system prompt (7 criteria, `Thoughts:`/`Status:` format) | `webjudge.py:89-113` | `FINAL_JUDGE_SYSTEM`, `prompts.ts:51` | verbatim ✓ | +| Stage 3 — final user text (`User Task / Key Points / Action History / snapshots+reasons`) | `webjudge.py:114-122,167-173` | `finalUserText`, `prompts.ts:88-108` | verbatim ✓ | +| Verdict parse `"success" in response.lower().split("status:")[1]` | `extract_predication`, `utils.py` (WebJudge mode) | `parseVerdict` → `raw.toLowerCase().split("status:")[1]?.includes("success")` (`prompts.ts:137`) | behaviorally identical ✓ | +| Order of the final image blocks (kept, in trajectory order, after the text) | `content: [{text}] + whole_content_img` (`webjudge.py:179-181`) | `[{text}, ...kept.map(image)]` (`webjudge.ts:39-55`) | ✓ | + +**Where the final verdict call lives.** Upstream `WebJudge_Online_Mind2Web_eval` +*returns* the assembled `messages`; `run.py` then makes the final +`model.generate(messages)` call and runs `extract_predication` (`run.py:104-114`). +Ours folds that final call inside `gradeWithWebJudge` (`webjudge.ts:56`). Same +two-call-then-verdict shape, just no function boundary. No behavioral difference. + +**`finalAnswer` is (correctly) not graded.** Upstream loads +`final_result_response`/`agent_final_answer` but **does not pass it to +`WebJudge_Online_Mind2Web_eval`** — only `WebVoyager_eval` consumes the final +answer (`run.py:89,99`). Our `gradeWithWebJudge` likewise never reads +`trajectory.finalAnswer`; the judge input is task + action history + scored +screenshots only. So the `answer.txt` plumbing is harmless ceremony and does not +affect grading — we match upstream here. (Note: the design doc's claim that "the +FINAL_JUDGE prompt consumes the final assistant text" is wrong; the code on both +sides does not, and that is the correct, faithful behavior.) + +--- + +## 2. Material differences + +### 2.1 Screenshots sent as PNG, upstream re-encodes to JPEG — `minor` (parity input) + +Upstream `encode_image` (`utils.py`) flattens RGBA→RGB and **re-encodes every +screenshot to JPEG** (`image.save(buffered, format="JPEG")`) before base64, +sending `data:image/jpeg;base64,...` with `"detail": "high"` to both the +per-image scorer and the final judge (`webjudge.py:62,79,154,158`). + +Ours reads the spilled PNG bytes from disk and sends them **as PNG** +(`screenshotMimeType ?? "image/png"`, `webjudge.ts:27,53`; `mimeForPath` keeps +the on-disk extension, `artifacts.ts:21-24`). The judge therefore sees the +original lossless PNG, not a JPEG-compressed, alpha-flattened copy. + +Impact: the bytes the model scores differ (JPEG artifacts + RGBA flatten vs +clean PNG). For a 1–5 "is this screenshot relevant" score and a coarse +success/failure verdict this rarely flips an outcome, so this is **minor**, not a +fidelity-bug — but it is a real divergence in judge input and is worth matching +if we want to reproduce a published number to the decimal. Suggested change: in +the artifacts loader, re-encode each screenshot to JPEG (RGBA→RGB) before +base64, mirroring `encode_image`, and send `image/jpeg`. (Anthropic accepts +`image/jpeg`; there is no `detail` field to set.) + +### 2.2 Judge backbone is Anthropic opus, not OpenAI o4-mini — `intentional-keep` (with a parity caveat) + +Upstream's documented recommendation is **o4-mini** ("please use o4-mini as the +backbone for automatic evaluation"; 85.7% human agreement, 3.8% success-rate +gap — README), `run.py` defaults to **gpt-4o**, and the shipped example results +use **gpt-4o-mini**. The judge is always an OpenAI chat model via `OpenaiEngine` +(`utils.py`). + +Ours uses `anthropic:claude-opus-4-8` (`task.toml:24`, `model.ts`), and +`model.ts` *only* wires the Anthropic provider — passing an `openai:...` +`JUDGE_MODEL` throws (`model.ts:39-43`). This is the deliberate Kernel +adaptation: a self-contained `fetch`-only bundle that runs in the browser VM with +no `npm install` and no OpenAI SDK, billed on the Anthropic key already present. +**Do not revert the Anthropic judge** — it is the point of the in-VM bundle. + +Parity caveat (document, don't revert): published Online-Mind2Web success rates +are calibrated to o4-mini/gpt-4o judges. An opus judge is a *different grader* +and will land at a different success rate, so our number is only comparable to a +baseline we recompute with the same judge — not to the paper's o4-mini column. +If decimal parity against the published o4-mini number is ever required, add an +OpenAI branch to `model.ts` (same `JudgeModel` interface, OpenAI +`/v1/chat/completions` with `image_url`+`detail:high`) and set +`JUDGE_MODEL=openai:o4-mini`; keep Anthropic as the default. Tracking this as a +parity *option*, not a bug. + +### 2.3 `max_tokens` 1024 vs upstream 512 — `minor` (kept, documented) + +Upstream calls `model.generate(..., max_new_tokens=512)` for all three stages +(key points, per-image, final — `utils.py` `generate` default, used everywhere). +Ours sends `max_tokens: 1024` on every Anthropic call (`model.ts`). + +Impact: only matters if a response would exceed 512 tokens. The final verdict is +`Thoughts: \nStatus: `; a long "Thoughts" could in +principle hit 512 and truncate **before** the `Status:` line, which upstream's +parser then reads as failure (`split("status:")[1]` → IndexError → 0). Ours at +1024 is less likely to truncate, so on a verbose trajectory we could score +success where upstream scores failure. Small and asymmetric; **minor**. + +**Resolution: keep 1024, document the headroom (done — `model.ts` comment).** +The opus judge is verbose: `JUDGE_IMAGE_SYSTEM` asks for "a detailed description +of the image…" before the `Score`, so a 512 cap could truncate that stage too +and silently zero a relevant screenshot. The smoke ran green at 1024 across 20 +live trials; dropping to 512 to match a budget calibrated for the terser +o4-mini/gpt-4o graders would risk regressing the judge we ship for no parity +gain (the judge model already diverges, §2.2). Deviation documented in code. + +### 2.4 `temperature` handling — `intentional-keep` + +Upstream hardcodes `temperature=0` for deterministic grading (`utils.py` +`OpenaiEngine.generate`). Ours also wants `temperature: 0`, but newer Anthropic +models (opus-4-8) reject the field with HTTP 400; `model.ts:73-80` sends +`temperature: 0` first and retries **once without it** on a 400 whose body +mentions `temperature`. This is required for the chosen judge to run at all (it +was a smoke-surfaced bug, SMOKE.md) and preserves determinism for models that +accept it. Keep. + +### 2.5 `difficulty` hardcoded; upstream `level` ignored — `minor` (metadata only) — FIXED + +Upstream rows carry a `level` field (`easy`/`medium`/`hard`; confirmed in the +live dataset — split: medium 141, easy 80, hard 79 across 300 rows). The adapter +previously hardcoded `difficulty = "hard"` in every `task.toml`, mislabeling +221/300 rows. **Fixed:** `parse_tasks` now reads `level`, `OnlineMind2WebTask` +carries it, and `_prepare_task` substitutes `{difficulty}` in the template +(default `"hard"` when absent) and writes `level` into `tests/task.json`. The +judge's `TaskJson` type carries the field but does not grade on it (metadata +only, like `reference_length`). Verified on real generation: easy/medium rows now +surface as `difficulty = "easy"`/`"medium"`. + +--- + +## 3. Dataset → task mapping (faithful) + +Upstream eval reads from per-task `result.json` submissions; the *task +definitions* live in the gated `osunlp/Online-Mind2Web` dataset. The live rows +have fields `task_id`, `confirmed_task`, `website`, `reference_length`, `level` +(verified against the 300-row cache). Our `parse_tasks` (`adapter.py:63`) maps: + +| Field | Upstream row | Our mapping | Match? | +|---|---|---|---| +| id | `task_id` | `task.id`; skip row if missing (`adapter.py:73-74`) | ✓ — matches the cua loader + `schema_v2` ("copied EXACTLY", hex hash) | +| instruction | `confirmed_task`, else `task` | `confirmed_task or task`; skip if both missing (`adapter.py:72-74`) | ✓ (fallback matches `dataset.ts`) | +| start URL | `website` | `start_url`: strip, prepend `https://` if no scheme, `None` if blank (`adapter.py:52-60`) | ✓ — needed because some rows are bare hosts (`apple.com`); blank → instruction-only prompt | +| reference_length | `reference_length` | carried to `task.json`/`[metadata]`, **not graded** (`adapter.py:82,187,196`) | ✓ — matches `schema_v2` ("efficiency-metric denominator, not graded") | +| level | `level` | `[metadata].difficulty` + `task.json.level`, default `hard` (see §2.5) | ✓ (fixed) | + +Skip-malformed (`adapter.py:73-74`) keeps the generated id set aligned with the +upstream set — the right invariant. Task count = 300 (one dir per surviving row). + +`make_local_task_id` (`adapter.py:135-143`) slugs the upstream id +(`lower`, `_`→`-`, strip non-`[a-z0-9-]`) for a Harbor-safe dir/name. This is a +*local* dir name; `tests/task.json` keeps the **raw** `task_id` +(`adapter.py:184`), which is what `schema_v2` requires graders to use ("Do NOT +add a prefix … raw ID only"). So the value the judge keys on stays canonical +even though the dir is slugged. ✓ + +--- + +## 4. Trajectory reconstruction (the new seam — faithful in spirit) + +Upstream consumes a `result.json` that already contains the action history and a +`trajectory/` dir of screenshots (`schema_v2`; `run.py:38-78`). We don't have +that artifact — the cua agent emits `run.jsonl` + spilled `shots/*.png` + +`answer.txt`. `loadTrajectory` (`artifacts.ts:56-102`) reconstructs the WebJudge +`Trajectory` from those: each `tool_result` line's `shots[]` becomes one step +whose **action string is the tool call that produced it** +(`actionString(name, arguments)`, `artifacts.ts:38-43,77-94`). + +Fidelity check against upstream's `last_actions` / `images_path`: + +- **Screenshot set.** Upstream scores *every* screenshot in the trajectory dir + (v1: all files sorted numerically; v2: every step's `screenshot`, + `run.py:50-78`). Ours scores every spilled shot, in run order. ✓ — no last-k + truncation on either side before scoring (the only cap is `MAX_IMAGE=50` on + *kept* images, §1). +- **Action strings.** Upstream's `last_actions` are the agent's own + free-text/grammar action strings from `result.json` (e.g. + `"CLICK coords(902,204) -> ... | SUCCESS"`, `schema_v2` example). Ours are + synthesized from the tool call (`" "`). Semantically the same + role (a numbered action history fed to the final judge), but the **surface form + differs** from a native Online-Mind2Web submission. This is inherent to driving + a different agent (cua computer-use) and is not a bug — the WebJudge prompt + asks for "the agent's action history" without prescribing a grammar. + Classification: acceptable adaptation, no change needed; flagged for awareness + when comparing transcripts side-by-side with upstream submissions. +- **Step/screenshot pairing.** `schema_v2` makes action↔screenshot a + per-step record to avoid v1 desync; ours pairs each shot to the `call_id` that + emitted it (`artifacts.ts:70-94`), preserving the same lock. ✓ + +--- + +## 5. Suggested changes (priority order) + +1. **(minor — skipped) JPEG-encode screenshots before the judge** to match + `encode_image` exactly (RGBA→RGB, `format="JPEG"`, `image/jpeg`). Closes the + last input-bytes gap with upstream, but needs an image codec in the + dependency-free in-VM bundle and buys no decimal parity while the judge is + opus (§2.2). Deferred — pair with item 4 if/when a published number is + targeted. — `artifacts.ts` (`loadTrajectory`) / `webjudge.ts` image blocks. +2. **(minor — kept, documented) `max_tokens`** left at 1024 with a comment + justifying the headroom; 512 would risk truncating the verbose opus judge. + — `model.ts`. *(Done.)* +3. **(minor — done) Carry `level` → `[metadata].difficulty`** instead of + hardcoding `"hard"`. Metadata only; enables per-difficulty breakdowns. + — `adapter.py` + `task.toml` + `task.json`. *(Done.)* +4. **(parity option, not a revert) Optional OpenAI judge branch** in `model.ts` + so `JUDGE_MODEL=openai:o4-mini` reproduces the published baseline when + needed; Anthropic stays the default. The in-VM Anthropic bundle is the + intended design and must not be removed. + +Item 3 is applied; items 1 and 2 are deferred/documented (see the applied-vs-skipped +note at the top); none are grading-correctness bugs. The WebJudge prompts, parsers, +thresholds, `MAX_IMAGE`, and verdict logic — the parts that decide success/failure — +already match the canonical source. + +## Source + +Official: `github.com/OSU-NLP-Group/Online-Mind2Web`, `main`, fetched +2026-06-27. Key files: `src/methods/webjudge_online_mind2web.py` (WebJudge), +`src/run.py` (driver/defaults: `--model gpt-4o`, `--score_threshold 3`), +`src/utils.py` (`encode_image` → JPEG; `extract_predication`; `OpenaiEngine` +`temperature=0`, `max_new_tokens=512`), `data/schema_v2/schema_v2.json` (task +fields), `README.md` ("use o4-mini for WebJudge"). diff --git a/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts b/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts index 278dac9..50e88cc 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/artifacts.ts @@ -8,6 +8,7 @@ interface TaskJson { instruction: string; start_url?: string | null; reference_length?: number | null; + level?: string | null; } const MIME_BY_EXT: Record = { diff --git a/benchmarks/adapters/online-mind2web/judge/src/model.ts b/benchmarks/adapters/online-mind2web/judge/src/model.ts index 0769b4f..ac380ae 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/model.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/model.ts @@ -58,6 +58,9 @@ export function anthropicJudgeModel(ref: string): JudgeModel { ): Promise { const body: Record = { model: name, + // Upstream caps each stage at 512 (max_new_tokens). We keep 1024 of headroom + // so the verbose opus judge's per-image reasoning isn't truncated before the + // `Score`/`Status:` line the parsers key on. max_tokens: 1024, system: systemPrompt, messages: [{ role: "user", content: toAnthropicContent(content) }], diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py b/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py index 0d581fd..00e17c8 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/adapter.py @@ -43,11 +43,19 @@ class OnlineMind2WebTask: """One Online-Mind2Web row mapped to the adapter's task shape.""" - def __init__(self, instruction: str, website: str | None, task_id: str, reference_length: int | None): + def __init__( + self, + instruction: str, + website: str | None, + task_id: str, + reference_length: int | None, + level: str | None = None, + ): self.id = task_id self.instruction = instruction self.website = website self.reference_length = reference_length + self.level = level @property def start_url(self) -> str | None: @@ -74,12 +82,14 @@ def parse_tasks(raw: str) -> list[OnlineMind2WebTask]: if not task_id or not instruction: continue ref_len = row.get("reference_length") + level = row.get("level") tasks.append( OnlineMind2WebTask( instruction=instruction, website=row.get("website"), task_id=task_id, reference_length=ref_len if isinstance(ref_len, int) else None, + level=level if isinstance(level, str) else None, ) ) return tasks @@ -185,16 +195,19 @@ def _prepare_task(self, task: OnlineMind2WebTask, output_dir: Path) -> None: "instruction": task.instruction, "start_url": task.start_url, "reference_length": task.reference_length, + "level": task.level, } (tests_dir / "task.json").write_text(json.dumps(task_json, indent=2) + "\n") - # task.toml — substitute the slugged id and reference length. + # task.toml — substitute the slugged id, reference length, and difficulty. task_toml = (TEMPLATE_DIR / "task.toml").read_text() task_toml = task_toml.replace("{task_id}", self.make_local_task_id(task.id)) task_toml = task_toml.replace( "{reference_length}", str(task.reference_length) if task.reference_length is not None else "", ) + # Carry the row's level (easy/medium/hard) through; default to hard when absent. + task_toml = task_toml.replace("{difficulty}", task.level or "hard") (output_dir / "task.toml").write_text(task_toml) # instruction.md — the agent prompt, with or without a start URL. diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml index 99f79a7..2ae51b6 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml @@ -6,7 +6,7 @@ authors = [{ name = "OSU NLP Group" }] keywords = ["online-mind2web", "web-agent", "live-web"] [metadata] -difficulty = "hard" +difficulty = "{difficulty}" category = "web-navigation" source = "osunlp/Online-Mind2Web" reference_length = "{reference_length}" diff --git a/benchmarks/adapters/online-mind2web/tests/test_adapter.py b/benchmarks/adapters/online-mind2web/tests/test_adapter.py index 771b7ce..e05021e 100644 --- a/benchmarks/adapters/online-mind2web/tests/test_adapter.py +++ b/benchmarks/adapters/online-mind2web/tests/test_adapter.py @@ -18,6 +18,7 @@ "confirmed_task": "Find the closest store to 90028.", "website": "https://www.traderjoes.com/", "reference_length": 6, + "level": "medium", }, # bare host -> normalized to https:// { @@ -105,10 +106,12 @@ def test_generates_full_task_dir(tmp_path): assert task_json["task_id"] == "abc123_110325" assert task_json["start_url"] == "https://www.traderjoes.com/" assert task_json["reference_length"] == 6 + assert task_json["level"] == "medium" toml = tomllib.loads((task_dir / "task.toml").read_text()) assert toml["task"]["name"] == "osunlp/online-mind2web-abc123-110325" assert toml["metadata"]["reference_length"] == "6" + assert toml["metadata"]["difficulty"] == "medium" assert toml["verifier"]["env"]["JUDGE_MODEL"].startswith("anthropic:") instruction = (task_dir / "instruction.md").read_text() @@ -126,6 +129,12 @@ def test_blank_website_omits_start_url_and_uses_nourl_prompt(tmp_path): assert "Go to" not in instruction assert "Do a thing." in instruction + # row carries no `level` -> difficulty defaults to hard, task.json level is null + toml = tomllib.loads((task_dir / "task.toml").read_text()) + assert toml["metadata"]["difficulty"] == "hard" + task_json = json.loads((task_dir / "tests/task.json").read_text()) + assert task_json["level"] is None + def test_limit_and_overwrite(tmp_path): out = _generate(tmp_path, limit=2) From 4d39c7dc4bd4b2e720b5dea91c592064ee78cfb0 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 19:15:03 +0000 Subject: [PATCH 4/9] online-mind2web: default WebJudge to OpenAI o4-mini Switch the judge backbone default from anthropic:claude-opus-4-8 to openai:o4-mini, the published WebJudge model, for leaderboard parity. Add a dependency-free OpenAI Chat Completions client to model.ts (stdlib fetch, no SDK) routed by the JUDGE_MODEL provider prefix. It supports vision (image_url data-URLs with detail:high) and handles the o-series reasoning quirks: omit temperature and use max_completion_tokens for o\d-prefixed names. The Anthropic Messages path stays wired as a configurable, non-canonical cheaper alternative. task.toml passes OPENAI_API_KEY (host-resolved) alongside ANTHROPIC_API_KEY and defaults JUDGE_MODEL to openai:o4-mini. README + PARITY.md document the new default, the configurable Anthropic path, and WebJudge-7B as a future GPU-hosted option. Mocked tests cover provider routing, the no-temperature / max_completion_tokens construction, and the vision payload shape. Co-Authored-By: Claude Opus 4.7 --- benchmarks/adapters/online-mind2web/PARITY.md | 164 ++++++++++-------- benchmarks/adapters/online-mind2web/README.md | 27 ++- .../online-mind2web/judge/package.json | 2 +- .../online-mind2web/judge/src/judge.ts | 11 +- .../online-mind2web/judge/src/model.ts | 115 ++++++++++-- .../judge/test/artifacts.test.ts | 140 ++++++++++++++- .../online_mind2web/task-template/task.toml | 7 +- .../task-template/tests/test.sh | 2 +- 8 files changed, 369 insertions(+), 99 deletions(-) diff --git a/benchmarks/adapters/online-mind2web/PARITY.md b/benchmarks/adapters/online-mind2web/PARITY.md index 7ad7b78..242e9a8 100644 --- a/benchmarks/adapters/online-mind2web/PARITY.md +++ b/benchmarks/adapters/online-mind2web/PARITY.md @@ -14,18 +14,23 @@ Our WebJudge is a TypeScript re-port bundled as a self-contained `judge.js` **Verdict: the WebJudge core is a faithful port.** The three system prompts, the user-text templates, key-point extraction, per-image 1–5 scoring, the `score_threshold` keep-filter, `MAX_IMAGE = 50`, and the `Status:` verdict parse -are all reproduced verbatim or behaviorally-identically. The differences that -matter are: (1) **the screenshots are sent as PNG, not JPEG** — upstream -re-encodes every image to JPEG before the judge sees it; (2) **the judge model -is Anthropic opus, not OpenAI o4-mini** — a deliberate Kernel choice that moves -us off the published parity baseline; (3) `max_tokens` is 1024 vs upstream's 512. -None of these change the *pipeline*; (1) and (2) change the *inputs the judge -scores*, so they bear on numeric parity. Details below. +are all reproduced verbatim or behaviorally-identically. The judge backbone now +defaults to **OpenAI o4-mini**, the published WebJudge model, so we sit on the +leaderboard-parity baseline. The remaining differences are: (1) **the screenshots +are sent as PNG, not JPEG** — upstream re-encodes every image to JPEG before the +judge sees it; (2) `max_tokens` is 1024 vs upstream's 512. Neither changes the +*pipeline*; (1) changes the *bytes the judge scores*, so it bears on decimal +numeric parity. Anthropic (opus/sonnet) stays wired as a configurable, +non-canonical cheaper alternative. Details below. ### Parity pass — applied vs skipped (2026-06-27) -A follow-up parity pass reviewed the three "minor" items in §2. Outcome: - +- **Applied — §2.2 default judge → OpenAI o4-mini.** The judge backbone now + defaults to `openai:o4-mini`, the published WebJudge model (~85.7% human + agreement), putting us on the leaderboard-parity baseline. `model.ts` gained a + dependency-free OpenAI Chat Completions client (vision `image_url` + + `detail: high`, o-series `max_completion_tokens` + no `temperature`); + Anthropic stays wired as a configurable, non-canonical alternative. - **Applied — §2.5 carry `level` → `[metadata].difficulty`.** The hardcoded `difficulty = "hard"` mislabeled 221/300 live rows (live `level` split: medium 141, easy 80, hard 79). The adapter now threads the row's `level` into @@ -33,14 +38,17 @@ A follow-up parity pass reviewed the three "minor" items in §2. Outcome: when absent. Metadata only — does not touch grading or selection. - **Skipped — §2.1 JPEG re-encode.** Requires an image codec (RGBA→RGB + JPEG) inside the judge, which conflicts with the deliberate self-contained, - dependency-free in-VM bundle, and buys no decimal parity while the judge model - is opus rather than the o4-mini/gpt-4o the published numbers are calibrated to - (§2.2). Not warranted now; revisit only alongside an OpenAI judge branch. + dependency-free in-VM bundle. Now that the judge is o4-mini this is the last + input-bytes gap on the parity path (PNG vs upstream's JPEG); still deferred — + it needs a codec in the bundle and rarely flips a coarse 1–5 / success + verdict — but revisit it first if decimal parity against a published number is + required. - **Skipped (documented, no behavior change) — §2.3 `max_tokens`.** Left at 1024 with a code comment explaining the headroom: lowering to upstream's 512 - risks truncating the verbose opus judge's per-image reasoning before the - `Score`/`Status:` line the parsers key on — a regression against the judge we - actually ship, with no parity payoff (opus ≠ o4-mini). + risks truncating a verbose per-image description before the `Score`/`Status:` + line the parsers key on. 1024 only matters when a response exceeds 512 tokens, + so it cannot truncate where upstream wouldn't — a safe superset, not a parity + risk. --- @@ -103,13 +111,14 @@ original lossless PNG, not a JPEG-compressed, alpha-flattened copy. Impact: the bytes the model scores differ (JPEG artifacts + RGBA flatten vs clean PNG). For a 1–5 "is this screenshot relevant" score and a coarse success/failure verdict this rarely flips an outcome, so this is **minor**, not a -fidelity-bug — but it is a real divergence in judge input and is worth matching -if we want to reproduce a published number to the decimal. Suggested change: in -the artifacts loader, re-encode each screenshot to JPEG (RGBA→RGB) before -base64, mirroring `encode_image`, and send `image/jpeg`. (Anthropic accepts -`image/jpeg`; there is no `detail` field to set.) +fidelity-bug — but with the judge now on o4-mini it is the *last* remaining input +divergence on the parity path, so it is the first thing to match if we want to +reproduce a published number to the decimal. Suggested change: in the artifacts +loader, re-encode each screenshot to JPEG (RGBA→RGB) before base64, mirroring +`encode_image`, and send `image/jpeg`. Our OpenAI client already sends +`detail: high`, matching upstream; only the encoding differs. -### 2.2 Judge backbone is Anthropic opus, not OpenAI o4-mini — `intentional-keep` (with a parity caveat) +### 2.2 Judge backbone defaults to OpenAI o4-mini — `matched` (Anthropic configurable) Upstream's documented recommendation is **o4-mini** ("please use o4-mini as the backbone for automatic evaluation"; 85.7% human agreement, 3.8% success-rate @@ -117,28 +126,37 @@ gap — README), `run.py` defaults to **gpt-4o**, and the shipped example result use **gpt-4o-mini**. The judge is always an OpenAI chat model via `OpenaiEngine` (`utils.py`). -Ours uses `anthropic:claude-opus-4-8` (`task.toml:24`, `model.ts`), and -`model.ts` *only* wires the Anthropic provider — passing an `openai:...` -`JUDGE_MODEL` throws (`model.ts:39-43`). This is the deliberate Kernel -adaptation: a self-contained `fetch`-only bundle that runs in the browser VM with -no `npm install` and no OpenAI SDK, billed on the Anthropic key already present. -**Do not revert the Anthropic judge** — it is the point of the in-VM bundle. - -Parity caveat (document, don't revert): published Online-Mind2Web success rates -are calibrated to o4-mini/gpt-4o judges. An opus judge is a *different grader* -and will land at a different success rate, so our number is only comparable to a -baseline we recompute with the same judge — not to the paper's o4-mini column. -If decimal parity against the published o4-mini number is ever required, add an -OpenAI branch to `model.ts` (same `JudgeModel` interface, OpenAI -`/v1/chat/completions` with `image_url`+`detail:high`) and set -`JUDGE_MODEL=openai:o4-mini`; keep Anthropic as the default. Tracking this as a -parity *option*, not a bug. +Ours now defaults to `openai:o4-mini` (`task.toml`, `judge.ts`). `model.ts` +dispatches on the `JUDGE_MODEL` provider prefix (`judgeModel`): `openai:` → +`openaiJudgeModel` (Chat Completions over `fetch`, `OPENAI_API_KEY`), +`anthropic:` → `anthropicJudgeModel` (Messages, `ANTHROPIC_API_KEY`). The OpenAI +client is dependency-free (stdlib `fetch`, no OpenAI SDK), so the in-VM bundle +property is preserved. It handles the o-series reasoning quirks: o4-mini rejects +`temperature` (omitted for o-series, like the opus 400-bug in §2.4) and uses +`max_completion_tokens` instead of `max_tokens`; screenshots are sent as vision +`image_url` data-URLs with `detail: high`, matching upstream's `encode_image` +call shape. + +This puts us on the published parity baseline: o4-mini is the grader the +~85.7%-agreement numbers are calibrated to, so a recomputed success rate is +directly comparable to the Online-Mind2Web leaderboard's o4-mini column (modulo +the PNG-vs-JPEG input gap, §2.1). + +Anthropic (`anthropic:claude-sonnet-4-6`, `anthropic:claude-opus-4-8`) remains +wired as a configurable, **non-canonical** cheaper alternative — switch by +setting `JUDGE_MODEL` only (both keys flow through `[verifier.env]`). An +opus/sonnet judge is a *different grader* and will land at a different success +rate, so a number graded by it is not comparable to the published o4-mini +column. **WebJudge-7B** (open weights, `osunlp/WebJudge-7B`) is a future cheaper +option but needs GPU hosting and so is not wired into the dependency-free in-VM +bundle. ### 2.3 `max_tokens` 1024 vs upstream 512 — `minor` (kept, documented) Upstream calls `model.generate(..., max_new_tokens=512)` for all three stages (key points, per-image, final — `utils.py` `generate` default, used everywhere). -Ours sends `max_tokens: 1024` on every Anthropic call (`model.ts`). +Ours sends `1024` on every call — `max_completion_tokens` for o-series, +`max_tokens` otherwise (`MAX_OUTPUT_TOKENS`, `model.ts`). Impact: only matters if a response would exceed 512 tokens. The final verdict is `Thoughts: \nStatus: `; a long "Thoughts" could in @@ -148,22 +166,29 @@ parser then reads as failure (`split("status:")[1]` → IndexError → 0). Ours success where upstream scores failure. Small and asymmetric; **minor**. **Resolution: keep 1024, document the headroom (done — `model.ts` comment).** -The opus judge is verbose: `JUDGE_IMAGE_SYSTEM` asks for "a detailed description -of the image…" before the `Score`, so a 512 cap could truncate that stage too -and silently zero a relevant screenshot. The smoke ran green at 1024 across 20 -live trials; dropping to 512 to match a budget calibrated for the terser -o4-mini/gpt-4o graders would risk regressing the judge we ship for no parity -gain (the judge model already diverges, §2.2). Deviation documented in code. +`JUDGE_IMAGE_SYSTEM` asks for "a detailed description of the image…" before the +`Score`, so a 512 cap could truncate that stage and silently zero a relevant +screenshot. 1024 is a safe superset of 512 — it cannot truncate where upstream +wouldn't — so it carries no parity risk against the o4-mini default while giving +the more verbose Anthropic path headroom too. Deviation documented in code. ### 2.4 `temperature` handling — `intentional-keep` Upstream hardcodes `temperature=0` for deterministic grading (`utils.py` -`OpenaiEngine.generate`). Ours also wants `temperature: 0`, but newer Anthropic -models (opus-4-8) reject the field with HTTP 400; `model.ts:73-80` sends -`temperature: 0` first and retries **once without it** on a 400 whose body -mentions `temperature`. This is required for the chosen judge to run at all (it -was a smoke-surfaced bug, SMOKE.md) and preserves determinism for models that -accept it. Keep. +`OpenaiEngine.generate`). Ours also wants `temperature: 0` but handles two model +families that reject it: + +- **OpenAI o-series (default o4-mini):** reasoning models reject `temperature` + outright, so the client omits it for any `o\d`-prefixed name (and uses + `max_completion_tokens`). Determinism still holds — o-series scoring is + effectively greedy at the default. +- **Anthropic (configurable):** newer models (opus-4-8) reject the field with + HTTP 400; the client sends `temperature: 0` first and retries **once without + it** on a 400 whose body mentions `temperature` (a smoke-surfaced bug, + SMOKE.md), preserving `temperature: 0` for models that accept it. + +Both paths converge on upstream's intent (deterministic grading) within each +provider's constraints. Keep. ### 2.5 `difficulty` hardcoded; upstream `level` ignored — `minor` (metadata only) — FIXED @@ -241,27 +266,28 @@ Fidelity check against upstream's `last_actions` / `images_path`: ## 5. Suggested changes (priority order) -1. **(minor — skipped) JPEG-encode screenshots before the judge** to match - `encode_image` exactly (RGBA→RGB, `format="JPEG"`, `image/jpeg`). Closes the - last input-bytes gap with upstream, but needs an image codec in the - dependency-free in-VM bundle and buys no decimal parity while the judge is - opus (§2.2). Deferred — pair with item 4 if/when a published number is - targeted. — `artifacts.ts` (`loadTrajectory`) / `webjudge.ts` image blocks. -2. **(minor — kept, documented) `max_tokens`** left at 1024 with a comment - justifying the headroom; 512 would risk truncating the verbose opus judge. - — `model.ts`. *(Done.)* -3. **(minor — done) Carry `level` → `[metadata].difficulty`** instead of +1. **(parity — done) Default judge → OpenAI o4-mini** with a dependency-free + `fetch` client in `model.ts`, routed by the `JUDGE_MODEL` provider prefix; + Anthropic stays configurable. Puts a recomputed success rate on the published + o4-mini baseline. — `model.ts` + `judge.ts` + `task.toml`. *(Done.)* +2. **(minor — skipped) JPEG-encode screenshots before the judge** to match + `encode_image` exactly (RGBA→RGB, `format="JPEG"`, `image/jpeg`). Now the last + input-bytes gap with upstream once the judge is o4-mini, but needs an image + codec in the dependency-free in-VM bundle. Deferred — do this first if/when + decimal parity against a published number is targeted. — `artifacts.ts` + (`loadTrajectory`) / `webjudge.ts` image blocks. +3. **(minor — kept, documented) `max_tokens`** left at 1024 with a comment + justifying the headroom; 512 would risk truncating a verbose per-image + description, and 1024 is a safe superset. — `model.ts`. *(Done.)* +4. **(minor — done) Carry `level` → `[metadata].difficulty`** instead of hardcoding `"hard"`. Metadata only; enables per-difficulty breakdowns. — `adapter.py` + `task.toml` + `task.json`. *(Done.)* -4. **(parity option, not a revert) Optional OpenAI judge branch** in `model.ts` - so `JUDGE_MODEL=openai:o4-mini` reproduces the published baseline when - needed; Anthropic stays the default. The in-VM Anthropic bundle is the - intended design and must not be removed. - -Item 3 is applied; items 1 and 2 are deferred/documented (see the applied-vs-skipped -note at the top); none are grading-correctness bugs. The WebJudge prompts, parsers, -thresholds, `MAX_IMAGE`, and verdict logic — the parts that decide success/failure — -already match the canonical source. + +Items 1, 3, and 4 are applied; item 2 is deferred/documented (see the +applied-vs-skipped note at the top); none are grading-correctness bugs. The +WebJudge prompts, parsers, thresholds, `MAX_IMAGE`, and verdict logic — the parts +that decide success/failure — already match the canonical source, and the judge +backbone now matches the published o4-mini recommendation. ## Source diff --git a/benchmarks/adapters/online-mind2web/README.md b/benchmarks/adapters/online-mind2web/README.md index 720b293..69df537 100644 --- a/benchmarks/adapters/online-mind2web/README.md +++ b/benchmarks/adapters/online-mind2web/README.md @@ -19,7 +19,7 @@ adapters/online-mind2web/ main.py CLI: --output-dir --limit --overwrite --task-ids task-template/ task.toml, instruction.md(+nourl), kernel.json, solve.sh, tests/test.sh judge/ self-contained WebJudge Node bin (bundled, runs in-VM) - src/ webjudge.ts + prompts.ts (recovered, tested) + model.ts (Anthropic) + judge.ts (CLI) + src/ webjudge.ts + prompts.ts (recovered, tested) + model.ts (OpenAI default, Anthropic configurable) + judge.ts (CLI) tests/test_adapter.py mocked adapter tests (no network) ``` @@ -42,10 +42,33 @@ Generated tasks land in `.tasks/` (gitignored). `tests/test.sh` runs the bundled `judge.js` inside the Kernel VM (which ships `node` + global `fetch`), reading the agent's artifacts under `/logs/agent` (`answer.txt`, `run.jsonl`, spilled `shots/*.png`), reconstructing the WebJudge -trajectory, grading it against an Anthropic judge model, and writing a single +trajectory, grading it against the configured judge model, and writing a single reward float to `/logs/verifier/reward.txt`. The judge model and score threshold are set in `[verifier.env]` (`JUDGE_MODEL`, `SCORE_THRESHOLD`). +### Judge backbone + +The default `JUDGE_MODEL` is **`openai:o4-mini`** — the published WebJudge +backbone (~85.7% human agreement), so a recomputed success rate is comparable to +the Online-Mind2Web leaderboard. The judge bin parses the provider from the +`JUDGE_MODEL` prefix and reads the matching key from the verifier env: + +- `openai:` → `OPENAI_API_KEY`. o-series models (o4-mini, o3, …) reject + `temperature` and use `max_completion_tokens`; the client omits/swaps these + automatically. Screenshots are sent as vision `image_url` blocks with + `detail: high`. +- `anthropic:` → `ANTHROPIC_API_KEY`. Configurable, **non-canonical** + cheaper alternative (e.g. `anthropic:claude-sonnet-4-6` / + `anthropic:claude-opus-4-8`); an opus/sonnet judge is a *different grader* and + will not match the published o4-mini numbers (see `PARITY.md` §2.2). + +Both keys are passed through `[verifier.env]`, resolved from host env, so either +provider works by changing only `JUDGE_MODEL`. + +[WebJudge-7B](https://huggingface.co/osunlp/WebJudge-7B) (open weights) is a +future cheaper option but needs GPU hosting, so it is not wired into the +dependency-free in-VM bundle. + ## Run on Harbor ```bash diff --git a/benchmarks/adapters/online-mind2web/judge/package.json b/benchmarks/adapters/online-mind2web/judge/package.json index 44890b9..2079044 100644 --- a/benchmarks/adapters/online-mind2web/judge/package.json +++ b/benchmarks/adapters/online-mind2web/judge/package.json @@ -1,7 +1,7 @@ { "name": "online-mind2web-webjudge", "version": "0.1.0", - "description": "Self-contained WebJudge verifier bin for the Online-Mind2Web Harbor adapter; runs in-VM via node, grades the agent trajectory against an Anthropic judge", + "description": "Self-contained WebJudge verifier bin for the Online-Mind2Web Harbor adapter; runs in-VM via node, grades the agent trajectory against the configured judge backbone (default OpenAI o4-mini; Anthropic configurable)", "license": "MIT", "private": true, "type": "module", diff --git a/benchmarks/adapters/online-mind2web/judge/src/judge.ts b/benchmarks/adapters/online-mind2web/judge/src/judge.ts index 348ae95..05f246b 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/judge.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/judge.ts @@ -4,8 +4,9 @@ * Runs inside the Kernel browser VM (which ships `node` + global `fetch`) as a * self-contained bundle — no `npm install` at verify time. Reads the agent's * artifacts under `/logs/agent` (answer.txt + run.jsonl + shots/), reconstructs - * the WebJudge trajectory, grades it with an Anthropic-backed judge, and writes - * a single reward float to `/logs/verifier/reward.txt`. + * the WebJudge trajectory, grades it with the configured judge backbone + * (default OpenAI o4-mini, the published WebJudge model), and writes a single + * reward float to `/logs/verifier/reward.txt`. * * A missing/empty trajectory or any failure writes reward `0` rather than * leaving the reward file empty (an empty reward is a Harbor verifier error). @@ -13,7 +14,7 @@ import { mkdirSync, writeFileSync } from "node:fs"; import { dirname } from "node:path"; import { loadTask, loadTrajectory } from "./artifacts.ts"; -import { anthropicJudgeModel } from "./model.ts"; +import { judgeModel } from "./model.ts"; import { gradeWithWebJudge } from "./webjudge.ts"; interface Args { @@ -46,7 +47,7 @@ function parseArgs(argv: string[]): Args { run: require("run"), answer: require("answer"), shots: flags.get("shots"), - judgeModel: flags.get("judge-model") ?? "anthropic:claude-opus-4-8", + judgeModel: flags.get("judge-model") ?? "openai:o4-mini", scoreThreshold: Number(flags.get("score-threshold") ?? "3"), rewardOut: require("reward-out"), detailsOut: flags.get("details-out"), @@ -67,7 +68,7 @@ async function main(): Promise { answerPath: args.answer, shotsBaseDir: args.shots, }); - const judge = anthropicJudgeModel(args.judgeModel); + const judge = judgeModel(args.judgeModel); const result = await gradeWithWebJudge({ task, trajectory, diff --git a/benchmarks/adapters/online-mind2web/judge/src/model.ts b/benchmarks/adapters/online-mind2web/judge/src/model.ts index ac380ae..13a59f6 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/model.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/model.ts @@ -1,16 +1,25 @@ import type { JudgeContent, JudgeModel } from "./types.ts"; /** - * A {@link JudgeModel} backed by the Anthropic Messages API over `fetch`. + * {@link JudgeModel} implementations backed by raw `fetch` calls to a provider's + * chat API. Self-contained so the bundled judge runs inside the Kernel browser + * VM with no `npm install` at verify time (the VM ships `node` and global + * `fetch` but not the cua SDKs or any provider SDK). * - * Self-contained so the bundled judge runs inside the Kernel browser VM with no - * `npm install` at verify time (the VM ships `node` and global `fetch` but not - * the cua SDKs). The judge model is given as a `provider:name` ref; only the - * `anthropic` provider is wired because it is the configured judge backbone. + * The judge model is given as a `provider:name` ref. `openai` is the default + * provider — canonical WebJudge runs on an OpenAI o-series backbone (o4-mini), + * which is what the published leaderboard numbers are calibrated to. `anthropic` + * stays wired as a configurable, non-canonical alternative. */ +const OPENAI_URL = "https://api.openai.com/v1/chat/completions"; const ANTHROPIC_URL = "https://api.anthropic.com/v1/messages"; const ANTHROPIC_VERSION = "2023-06-01"; +// Upstream caps each stage at 512 (max_new_tokens). We keep 1024 of headroom so +// the verbose judge's per-image reasoning isn't truncated before the +// `Score`/`Status:` line the parsers key on. +const MAX_OUTPUT_TOKENS = 1024; + interface AnthropicBlock { type: string; text?: string; @@ -18,10 +27,93 @@ interface AnthropicBlock { export function parseModelRef(ref: string): { provider: string; name: string } { const idx = ref.indexOf(":"); - if (idx === -1) return { provider: "anthropic", name: ref }; + if (idx === -1) return { provider: "openai", name: ref }; return { provider: ref.slice(0, idx), name: ref.slice(idx + 1) }; } +/** Provider dispatch for `JUDGE_MODEL` refs. Defaults to OpenAI when no prefix. */ +export function judgeModel(ref: string): JudgeModel { + const { provider } = parseModelRef(ref); + switch (provider) { + case "openai": + return openaiJudgeModel(ref); + case "anthropic": + return anthropicJudgeModel(ref); + default: + throw new Error( + `unsupported judge provider "${provider}" (openai or anthropic); got "${ref}"`, + ); + } +} + +/** o-series reasoning models (o4-mini, o3, …) reject `temperature` and need + * `max_completion_tokens` instead of `max_tokens`. */ +function isOSeries(name: string): boolean { + return /^o\d/.test(name); +} + +function toOpenAIContent(content: JudgeContent): unknown[] { + return content.map((part) => + part.type === "text" + ? { type: "text", text: part.text } + : { + type: "image_url", + // WebJudge sends `detail: high` so the judge scores the full screenshot. + image_url: { + url: `data:${part.mimeType};base64,${part.data}`, + detail: "high", + }, + }, + ); +} + +/** OpenAI Chat Completions judge. Resolves the API key from `OPENAI_API_KEY`. */ +export function openaiJudgeModel(ref: string): JudgeModel { + const { provider, name } = parseModelRef(ref); + if (provider !== "openai") { + throw new Error(`openaiJudgeModel got a non-openai ref "${ref}"`); + } + const apiKey = process.env.OPENAI_API_KEY; + if (!apiKey) { + throw new Error("OPENAI_API_KEY is required for the WebJudge model"); + } + const oSeries = isOSeries(name); + const headers = { + "content-type": "application/json", + authorization: `Bearer ${apiKey}`, + }; + + return { + async complete(systemPrompt, content) { + const body: Record = { + model: name, + // o-series uses max_completion_tokens; older chat models use max_tokens. + [oSeries ? "max_completion_tokens" : "max_tokens"]: MAX_OUTPUT_TOKENS, + messages: [ + { role: "system", content: systemPrompt }, + { role: "user", content: toOpenAIContent(content) }, + ], + }; + // WebJudge wants deterministic scoring (temperature 0), but o-series + // reasoning models reject the field outright — omit it for o-series. + if (!oSeries) body.temperature = 0; + const res = await fetch(OPENAI_URL, { + method: "POST", + headers, + body: JSON.stringify(body), + }); + if (!res.ok) { + const errBody = await res.text(); + throw new Error(`OpenAI API error ${res.status}: ${errBody}`); + } + const data = (await res.json()) as { + choices?: Array<{ message?: { content?: string } }>; + }; + return data.choices?.[0]?.message?.content ?? ""; + }, + }; +} + function toAnthropicContent(content: JudgeContent): unknown[] { return content.map((part) => part.type === "text" @@ -33,13 +125,11 @@ function toAnthropicContent(content: JudgeContent): unknown[] { ); } -/** Anthropic-backed judge. Resolves the API key from `ANTHROPIC_API_KEY`. */ +/** Anthropic Messages judge. Resolves the API key from `ANTHROPIC_API_KEY`. */ export function anthropicJudgeModel(ref: string): JudgeModel { const { provider, name } = parseModelRef(ref); if (provider !== "anthropic") { - throw new Error( - `unsupported judge provider "${provider}" (only anthropic is wired); got "${ref}"`, - ); + throw new Error(`anthropicJudgeModel got a non-anthropic ref "${ref}"`); } const apiKey = process.env.ANTHROPIC_API_KEY; if (!apiKey) { @@ -58,10 +148,7 @@ export function anthropicJudgeModel(ref: string): JudgeModel { ): Promise { const body: Record = { model: name, - // Upstream caps each stage at 512 (max_new_tokens). We keep 1024 of headroom - // so the verbose opus judge's per-image reasoning isn't truncated before the - // `Score`/`Status:` line the parsers key on. - max_tokens: 1024, + max_tokens: MAX_OUTPUT_TOKENS, system: systemPrompt, messages: [{ role: "user", content: toAnthropicContent(content) }], }; diff --git a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts index 3b2ae0e..29047d7 100644 --- a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts +++ b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts @@ -4,7 +4,13 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; import { loadTask, loadTrajectory } from "../src/artifacts.ts"; -import { anthropicJudgeModel, parseModelRef } from "../src/model.ts"; +import { + anthropicJudgeModel, + judgeModel, + openaiJudgeModel, + parseModelRef, +} from "../src/model.ts"; +import type { JudgeContent } from "../src/types.ts"; // 1x1 transparent PNG, the same fixture the shared-core sink test uses. const PNG_B64 = @@ -123,10 +129,10 @@ describe("parseModelRef", () => { }); }); - it("defaults provider to anthropic when no colon", () => { - expect(parseModelRef("claude-sonnet-4-6")).toEqual({ - provider: "anthropic", - name: "claude-sonnet-4-6", + it("defaults provider to openai when no colon", () => { + expect(parseModelRef("o4-mini")).toEqual({ + provider: "openai", + name: "o4-mini", }); }); }); @@ -178,3 +184,127 @@ describe("anthropicJudgeModel", () => { ).rejects.toThrow("Anthropic API error 400"); }); }); + +describe("openaiJudgeModel", () => { + afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); + }); + + function okResponse(text: string): Response { + return new Response(JSON.stringify({ choices: [{ message: { content: text } }] }), { + status: 200, + }); + } + + /** Stub fetch with an OK response and capture each request body. */ + function captureBodies(text = "out"): Array> { + const bodies: Array> = []; + vi.stubGlobal( + "fetch", + vi.fn(async (_url: string, init: RequestInit) => { + bodies.push(JSON.parse(init.body as string)); + return okResponse(text); + }), + ); + return bodies; + } + + const vision: JudgeContent = [ + { type: "text", text: "score this" }, + { type: "image", data: "AAAA", mimeType: "image/png" }, + ]; + + it("omits temperature and uses max_completion_tokens for o-series", async () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + const bodies = captureBodies(); + + const out = await openaiJudgeModel("openai:o4-mini").complete("sys", [ + { type: "text", text: "hi" }, + ]); + + expect(out).toBe("out"); + expect(bodies[0]).not.toHaveProperty("temperature"); + expect(bodies[0].max_completion_tokens).toBe(1024); + expect(bodies[0]).not.toHaveProperty("max_tokens"); + expect(bodies[0].model).toBe("o4-mini"); + }); + + it("sends temperature 0 and max_tokens for non-o-series chat models", async () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + const bodies = captureBodies(); + + await openaiJudgeModel("openai:gpt-4o").complete("sys", [{ type: "text", text: "hi" }]); + + expect(bodies[0].temperature).toBe(0); + expect(bodies[0].max_tokens).toBe(1024); + expect(bodies[0]).not.toHaveProperty("max_completion_tokens"); + }); + + it("encodes images as data-url image_url with detail:high", async () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + const bodies = captureBodies(); + + await openaiJudgeModel("openai:o4-mini").complete("sys", vision); + + const messages = bodies[0].messages as Array<{ role: string; content: unknown }>; + expect(messages[0]).toEqual({ role: "system", content: "sys" }); + expect(messages[1].content).toEqual([ + { type: "text", text: "score this" }, + { + type: "image_url", + image_url: { url: "data:image/png;base64,AAAA", detail: "high" }, + }, + ]); + }); + + it("reads the assistant content from the first choice", async () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + captureBodies("Status: success"); + + const out = await openaiJudgeModel("openai:o4-mini").complete("sys", vision); + expect(out).toBe("Status: success"); + }); + + it("throws when the API key is missing", () => { + vi.stubEnv("OPENAI_API_KEY", ""); + expect(() => openaiJudgeModel("openai:o4-mini")).toThrow("OPENAI_API_KEY is required"); + }); + + it("throws on a non-2xx response", async () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + vi.stubGlobal( + "fetch", + vi.fn(async () => new Response(JSON.stringify({ error: { message: "rate limit" } }), { status: 429 })), + ); + + await expect( + openaiJudgeModel("openai:o4-mini").complete("sys", [{ type: "text", text: "hi" }]), + ).rejects.toThrow("OpenAI API error 429"); + }); +}); + +describe("judgeModel routing", () => { + afterEach(() => { + vi.unstubAllEnvs(); + }); + + it("routes openai: refs to the OpenAI client", () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + expect(() => judgeModel("openai:o4-mini")).not.toThrow(); + }); + + it("routes a bare ref to OpenAI (default provider)", () => { + vi.stubEnv("OPENAI_API_KEY", "k"); + expect(() => judgeModel("o4-mini")).not.toThrow(); + }); + + it("routes anthropic: refs to the Anthropic client", () => { + vi.stubEnv("ANTHROPIC_API_KEY", "k"); + expect(() => judgeModel("anthropic:claude-opus-4-8")).not.toThrow(); + }); + + it("throws on an unsupported provider", () => { + expect(() => judgeModel("gemini:flash")).toThrow('unsupported judge provider "gemini"'); + }); +}); diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml index 2ae51b6..0bff02d 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/task.toml @@ -19,9 +19,12 @@ timeout_sec = 900.0 [verifier.env] # WebJudge backbone. The judge bin parses the provider from JUDGE_MODEL and -# reads the matching provider key; anthropic is the wired provider. +# reads the matching provider key. Default openai:o4-mini is the published +# WebJudge backbone (leaderboard parity); set JUDGE_MODEL=anthropic: +# to use the Anthropic path instead (non-canonical, cheaper alternative). +OPENAI_API_KEY = "${OPENAI_API_KEY}" ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" -JUDGE_MODEL = "anthropic:claude-opus-4-8" +JUDGE_MODEL = "openai:o4-mini" SCORE_THRESHOLD = "3" [environment] diff --git a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh index 2558091..797c3d7 100644 --- a/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh +++ b/benchmarks/adapters/online-mind2web/src/online_mind2web/task-template/tests/test.sh @@ -13,7 +13,7 @@ node /tests/judge.js \ --run /logs/agent/run.jsonl \ --answer /logs/agent/answer.txt \ --shots /logs/agent \ - --judge-model "${JUDGE_MODEL:-anthropic:claude-opus-4-8}" \ + --judge-model "${JUDGE_MODEL:-openai:o4-mini}" \ --score-threshold "${SCORE_THRESHOLD:-3}" \ --reward-out /logs/verifier/reward.txt \ --details-out /logs/verifier/grading_details.json From abd180475ad4b3f4fffe8ef1b4dbe58bf4c4ea66 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 21:35:27 +0000 Subject: [PATCH 5/9] online-mind2web: back WebJudge model with pi-ai Replace the hand-rolled raw-fetch OpenAI/Anthropic judge clients with @earendil-works/pi-ai's completeSimple, which owns provider routing, env-key resolution, the o-series reasoning quirks, and vision encoding. Resolve the JUDGE_MODEL ref (default openai:o4-mini) to a pi-ai model and surface pi-ai's stopReason "error" as a throw so a grading failure is logged rather than scored as a task failure. Bundle pi-ai via tsdown (inlineDynamicImports) so the verifier stays a single self-contained judge.js. Prompts, parsing, scoring, and the model default are unchanged. Co-Authored-By: Claude Opus 4.7 --- .../online-mind2web/judge/package-lock.json | 1186 ++++++++++++++++- .../online-mind2web/judge/package.json | 3 + .../online-mind2web/judge/src/model.ts | 189 +-- .../judge/test/artifacts.test.ts | 237 ++-- .../online-mind2web/judge/tsdown.config.ts | 6 +- 5 files changed, 1313 insertions(+), 308 deletions(-) diff --git a/benchmarks/adapters/online-mind2web/judge/package-lock.json b/benchmarks/adapters/online-mind2web/judge/package-lock.json index c5679c3..7a63464 100644 --- a/benchmarks/adapters/online-mind2web/judge/package-lock.json +++ b/benchmarks/adapters/online-mind2web/judge/package-lock.json @@ -8,6 +8,9 @@ "name": "online-mind2web-webjudge", "version": "0.1.0", "license": "MIT", + "dependencies": { + "@earendil-works/pi-ai": "^0.79.1" + }, "bin": { "online-mind2web-webjudge": "dist/judge.js" }, @@ -21,6 +24,474 @@ "node": ">=22.0.0" } }, + "node_modules/@anthropic-ai/sdk": { + "version": "0.91.1", + "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.91.1.tgz", + "integrity": "sha512-LAmu761tSN9r66ixvmciswUj/ZC+1Q4iAfpedTfSVLeswRwnY3n2Nb6Tsk+cLPP28aLOPWeMgIuTuCcMC6W/iw==", + "license": "MIT", + "dependencies": { + "json-schema-to-ts": "^3.1.1" + }, + "bin": { + "anthropic-ai-sdk": "bin/cli" + }, + "peerDependencies": { + "zod": "^3.25.0 || ^4.0.0" + }, + "peerDependenciesMeta": { + "zod": { + "optional": true + } + } + }, + "node_modules/@aws-crypto/crc32": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/crc32/-/crc32-5.2.0.tgz", + "integrity": "sha512-nLbCWqQNgUiwwtFsen1AdzAtvuLRsQS8rYgMuxCrdKf9kOssamGLuPwyTY9wyYblNr9+1XM8v6zoDTPPSIeANg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/util": "^5.2.0", + "@aws-sdk/types": "^3.222.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=16.0.0" + } + }, + "node_modules/@aws-crypto/sha256-browser": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/sha256-browser/-/sha256-browser-5.2.0.tgz", + "integrity": "sha512-AXfN/lGotSQwu6HNcEsIASo7kWXZ5HYWvfOmSNKDsEqC4OashTp8alTmaz+F7TC2L083SFv5RdB+qU3Vs1kZqw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-js": "^5.2.0", + "@aws-crypto/supports-web-crypto": "^5.2.0", + "@aws-crypto/util": "^5.2.0", + "@aws-sdk/types": "^3.222.0", + "@aws-sdk/util-locate-window": "^3.0.0", + "@smithy/util-utf8": "^2.0.0", + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-crypto/sha256-js": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/sha256-js/-/sha256-js-5.2.0.tgz", + "integrity": "sha512-FFQQyu7edu4ufvIZ+OadFpHHOt+eSTBaYaki44c+akjg7qZg9oOQeLlk77F6tSYqjDAFClrHJk9tMf0HdVyOvA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/util": "^5.2.0", + "@aws-sdk/types": "^3.222.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=16.0.0" + } + }, + "node_modules/@aws-crypto/supports-web-crypto": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/supports-web-crypto/-/supports-web-crypto-5.2.0.tgz", + "integrity": "sha512-iAvUotm021kM33eCdNfwIN//F77/IADDSs58i+MDaOqFrVjZo9bAal0NK7HurRuWLLpF1iLX7gbWrjHjeo+YFg==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-crypto/util": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/util/-/util-5.2.0.tgz", + "integrity": "sha512-4RkU9EsI6ZpBve5fseQlGNUWKMa1RLPQ1dnjnQoe07ldfIzcsGb5hC5W0Dm7u423KWzawlrpbjXBrXCEv9zazQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.222.0", + "@smithy/util-utf8": "^2.0.0", + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-sdk/client-bedrock-runtime": { + "version": "3.1048.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/client-bedrock-runtime/-/client-bedrock-runtime-3.1048.0.tgz", + "integrity": "sha512-u+NT61JZEkRFtpL0CAw1N1dwxnaLgwVXQl/zjJxTGgLyS/jTIdg2SdoEoCTHxgDyCnqa1HEi9QOoE9/pYRNpOQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-browser": "5.2.0", + "@aws-crypto/sha256-js": "5.2.0", + "@aws-sdk/core": "^3.974.11", + "@aws-sdk/credential-provider-node": "^3.972.42", + "@aws-sdk/eventstream-handler-node": "^3.972.16", + "@aws-sdk/middleware-eventstream": "^3.972.12", + "@aws-sdk/middleware-websocket": "^3.972.19", + "@aws-sdk/token-providers": "3.1048.0", + "@aws-sdk/types": "^3.973.8", + "@smithy/core": "^3.24.2", + "@smithy/fetch-http-handler": "^5.4.2", + "@smithy/node-http-handler": "^4.7.2", + "@smithy/types": "^4.14.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/core": { + "version": "3.974.23", + "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.974.23.tgz", + "integrity": "sha512-MiWR/uWjxjFXGzrE0Ghc5lWxUxzHsUWFhV+OX7M4cR9SrmrnZs6TXavnCWnzzdwJeFri34xQo81rvGNzK3c4BQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.973.13", + "@aws-sdk/xml-builder": "^3.972.31", + "@aws/lambda-invoke-store": "^0.2.2", + "@smithy/core": "^3.24.6", + "@smithy/signature-v4": "^5.4.6", + "@smithy/types": "^4.14.3", + "bowser": "^2.11.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-env": { + "version": "3.972.49", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.49.tgz", + "integrity": "sha512-liB3yQNHCM9k/gu/w36XHMKPluT7HTlnGUhRbBGSISDQkcr/Sy1zsZabiuvQj8WG5yW573u9RehrBvvnIQ9OEQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-http": { + "version": "3.972.51", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.51.tgz", + "integrity": "sha512-XET0H2oofciJ5lMRWNIvRjAP7Q3wv2XT+JtJJEdhPWUMwe3TvQ9qcxonpu7vXmNngncvFpi4E2It+Tamas/naA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/fetch-http-handler": "^5.4.6", + "@smithy/node-http-handler": "^4.7.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-http/node_modules/@smithy/node-http-handler": { + "version": "4.8.2", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.8.2.tgz", + "integrity": "sha512-wfl1uwrAqMH9/pi4kqBo5LBcFwrJLxuDLqL7p7qNcJIFcyZDUc6pzhYk4CYv+DP7fIUpQCZumwNnkhPKS52osQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.26.0", + "@smithy/types": "^4.15.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-ini": { + "version": "3.972.56", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.972.56.tgz", + "integrity": "sha512-IAmc61hbgQiHht9U3x0tnRwz0lzdwOwD/i9voRgdJrKamF+JtmrBOsW9GwB7mfFonNWOWL4qARWYrF8veEMe3w==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/credential-provider-env": "^3.972.49", + "@aws-sdk/credential-provider-http": "^3.972.51", + "@aws-sdk/credential-provider-login": "^3.972.55", + "@aws-sdk/credential-provider-process": "^3.972.49", + "@aws-sdk/credential-provider-sso": "^3.972.55", + "@aws-sdk/credential-provider-web-identity": "^3.972.55", + "@aws-sdk/nested-clients": "^3.997.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/credential-provider-imds": "^4.3.7", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-login": { + "version": "3.972.55", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.55.tgz", + "integrity": "sha512-hBBkANo3cDn+h2qxxzER4a+J8JCO9o9Z/YYmU7iky6AcaarX5RRdRcHNC6SLdwY0vAXQygn6soUbDqPn3GghaA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/nested-clients": "^3.997.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-node": { + "version": "3.972.58", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.58.tgz", + "integrity": "sha512-OyCLVmSI7pZO8hxwNVX6pXhTVlJqRBTp+ijdEfJSUj0RyjHnF602OfAarOzGq6wkGodeFkYBt8MmJ6A6ycRgWw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/credential-provider-env": "^3.972.49", + "@aws-sdk/credential-provider-http": "^3.972.51", + "@aws-sdk/credential-provider-ini": "^3.972.56", + "@aws-sdk/credential-provider-process": "^3.972.49", + "@aws-sdk/credential-provider-sso": "^3.972.55", + "@aws-sdk/credential-provider-web-identity": "^3.972.55", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/credential-provider-imds": "^4.3.7", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-process": { + "version": "3.972.49", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.49.tgz", + "integrity": "sha512-C8h36lBuC/RnBSsjlO+dn6xZm3KbAl5vpJaVPAfQnMmz2/OISmKOc8XZcqMQgO2ADwBYNRMM6Kf3vz9G/TulMQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-sso": { + "version": "3.972.55", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.972.55.tgz", + "integrity": "sha512-1FkOz74Ea5QGS9jtIoXp55T/IkSS3spv+nLTT07fRY/+T5xmEOqaYBVIaEmX4zTNvbV6g2lrtlaVKWEoNyJt3w==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/nested-clients": "^3.997.23", + "@aws-sdk/token-providers": "3.1074.0", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-sso/node_modules/@aws-sdk/token-providers": { + "version": "3.1074.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1074.0.tgz", + "integrity": "sha512-pv80IzgGW4RnXWtft692chZOM9i6PhebVsLCcnaM4dBEPZva2fE6FXAHs76G7Rc7s3yGyX/68G0nZMrUy+Vmpg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/nested-clients": "^3.997.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-web-identity": { + "version": "3.972.55", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.55.tgz", + "integrity": "sha512-g2BoECD1q01kTPByi56+VLVvdWDzMkKIcr77qixpqH0okw2t0U5CoPv+6S8v/D1Y2Wa6QKKtn6XAtDzP+Kfpvg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/nested-clients": "^3.997.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/eventstream-handler-node": { + "version": "3.972.22", + "resolved": "https://registry.npmjs.org/@aws-sdk/eventstream-handler-node/-/eventstream-handler-node-3.972.22.tgz", + "integrity": "sha512-tqPJv0dz4+O0hWGm1a6YekcMZyPhDFs/zH73Von7icaVT5n0Jqvm86typ3jRrG+qoUdPhALOnboRLTmnWQTlYQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/middleware-eventstream": { + "version": "3.972.18", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-eventstream/-/middleware-eventstream-3.972.18.tgz", + "integrity": "sha512-OHpk8YoZi3yexPq8aFt1vN1IxA2zLKvsIR5GpWYylX/ve6kQmY7wxHNSFy/D3t2apMZ16rs76Co4dJWcDyIk3A==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/middleware-websocket": { + "version": "3.972.31", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-websocket/-/middleware-websocket-3.972.31.tgz", + "integrity": "sha512-ps1rumU1LybSFHaW9dTDgkhCMJLVaedEY78kKSzUDDY+b9974/g6aiaYYA0U9WV0oL4CJCJrVWG+EZ/qr4or7g==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/fetch-http-handler": "^5.4.6", + "@smithy/signature-v4": "^5.4.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@aws-sdk/nested-clients": { + "version": "3.997.23", + "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.23.tgz", + "integrity": "sha512-gO93ZPsI2bxeFZD42f1/qjDw6FAZkNZcKRO94LIiT03fzOmcJ9e/tunxjVjA1Rl69ClmVJzz8H3G9CdKef10PA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-browser": "5.2.0", + "@aws-crypto/sha256-js": "5.2.0", + "@aws-sdk/core": "^3.974.23", + "@aws-sdk/signature-v4-multi-region": "^3.996.35", + "@aws-sdk/types": "^3.973.13", + "@smithy/core": "^3.24.6", + "@smithy/fetch-http-handler": "^5.4.6", + "@smithy/node-http-handler": "^4.7.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/nested-clients/node_modules/@smithy/node-http-handler": { + "version": "4.8.2", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.8.2.tgz", + "integrity": "sha512-wfl1uwrAqMH9/pi4kqBo5LBcFwrJLxuDLqL7p7qNcJIFcyZDUc6pzhYk4CYv+DP7fIUpQCZumwNnkhPKS52osQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.26.0", + "@smithy/types": "^4.15.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/signature-v4-multi-region": { + "version": "3.996.35", + "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.35.tgz", + "integrity": "sha512-6L/VWs+Wch2stHemCGTmUNqKLMzURxQDK5boNG3Jn3kAOp71meDUuS5sbObpEvFxHDq0uWeSLFDNSYsjNt+Dlg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.973.13", + "@smithy/signature-v4": "^5.4.6", + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/token-providers": { + "version": "3.1048.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1048.0.tgz", + "integrity": "sha512-k0y/GcuesuSfWyUM0WamrGyeZmltRYaPbHO82UDA6mZ/doB+FOHKutikPAtSXMn/hDz970cF+iRuuiYO9VEbAA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.974.11", + "@aws-sdk/nested-clients": "^3.997.9", + "@aws-sdk/types": "^3.973.8", + "@smithy/core": "^3.24.2", + "@smithy/types": "^4.14.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/types": { + "version": "3.973.13", + "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.973.13.tgz", + "integrity": "sha512-pEHZqRkAlHfnfAU9tK+WpKv/gBNjGJrHMgA3A0iYRGyswBS2t0pfez+lWlwktb3Bqa0ovh7w/QJTFwp3fDxLNg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/util-locate-window": { + "version": "3.965.8", + "resolved": "https://registry.npmjs.org/@aws-sdk/util-locate-window/-/util-locate-window-3.965.8.tgz", + "integrity": "sha512-uUbMs1cBZPafD0ohUj6EwNf0fPZ534NvBxHox4hjX+0Rxq5paSYUem7+hi833pYrzrcnBATKIYpR02MDXT5M9g==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/xml-builder": { + "version": "3.972.31", + "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.31.tgz", + "integrity": "sha512-SzE4Pgyl+hDF+BuyuzxUSpwnuUu9lJuO1YGgteG89/4Qv0+2IQiVQqdbPV32IozLvXWQChPQcdkk/sKvb1QHiQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.14.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws/lambda-invoke-store": { + "version": "0.2.4", + "resolved": "https://registry.npmjs.org/@aws/lambda-invoke-store/-/lambda-invoke-store-0.2.4.tgz", + "integrity": "sha512-iY8yvjE0y651BixKNPgmv1WrQc+GZ142sb0z4gYnChDDY2YqI4P/jsSopBWrKfAt7LOJAkOXt7rC/hms+WclQQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/@babel/generator": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-8.0.0.tgz", @@ -75,6 +546,15 @@ "node": "^22.18.0 || >=24.11.0" } }, + "node_modules/@babel/runtime": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.7.tgz", + "integrity": "sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==", + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, "node_modules/@babel/types": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/@babel/types/-/types-8.0.0.tgz", @@ -89,6 +569,31 @@ "node": "^22.18.0 || >=24.11.0" } }, + "node_modules/@earendil-works/pi-ai": { + "version": "0.79.10", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-ai/-/pi-ai-0.79.10.tgz", + "integrity": "sha512-9jR23tOl0BIUdQMn70Gr72xYBpM7Xgl9Lyv7gAnU1USfkNRuYG/f/edLl+n/Dp/RafDW3JI4DF7y/GhgkORuew==", + "license": "MIT", + "dependencies": { + "@anthropic-ai/sdk": "0.91.1", + "@aws-sdk/client-bedrock-runtime": "3.1048.0", + "@google/genai": "1.52.0", + "@mistralai/mistralai": "2.2.6", + "@opentelemetry/api": "1.9.0", + "@smithy/node-http-handler": "4.7.3", + "http-proxy-agent": "7.0.2", + "https-proxy-agent": "7.0.6", + "openai": "6.26.0", + "partial-json": "0.1.7", + "typebox": "1.1.38" + }, + "bin": { + "pi-ai": "dist/cli.js" + }, + "engines": { + "node": ">=22.19.0" + } + }, "node_modules/@emnapi/core": { "version": "1.11.1", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", @@ -565,6 +1070,30 @@ "node": ">=18" } }, + "node_modules/@google/genai": { + "version": "1.52.0", + "resolved": "https://registry.npmjs.org/@google/genai/-/genai-1.52.0.tgz", + "integrity": "sha512-gwSvbpiN/17O9TbsqSsE/OzZcpv5Fo4RQjdngGgogtuB9RsyJ8ZHhX5KjHj1bp5N9snN2eK8LDGXSaWW2hof8Q==", + "hasInstallScript": true, + "license": "Apache-2.0", + "dependencies": { + "google-auth-library": "^10.3.0", + "p-retry": "^4.6.2", + "protobufjs": "^7.5.4", + "ws": "^8.18.0" + }, + "engines": { + "node": ">=20.0.0" + }, + "peerDependencies": { + "@modelcontextprotocol/sdk": "^1.25.2" + }, + "peerDependenciesMeta": { + "@modelcontextprotocol/sdk": { + "optional": true + } + } + }, "node_modules/@jridgewell/gen-mapping": { "version": "0.3.13", "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", @@ -604,6 +1133,26 @@ "@jridgewell/sourcemap-codec": "^1.4.14" } }, + "node_modules/@mistralai/mistralai": { + "version": "2.2.6", + "resolved": "https://registry.npmjs.org/@mistralai/mistralai/-/mistralai-2.2.6.tgz", + "integrity": "sha512-W8pX7zHxjJvMIpw8JMxeJEleapXX0Q9NPszdNzqkM3MIEoIGPObdodujj+WHteXEvGfaP/AMwlNyRfEzSY6dQQ==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/semantic-conventions": "^1.40.0", + "ws": "^8.18.0", + "zod": "^3.25.0 || ^4.0.0", + "zod-to-json-schema": "^3.25.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.9.0" + }, + "peerDependenciesMeta": { + "@opentelemetry/api": { + "optional": true + } + } + }, "node_modules/@napi-rs/wasm-runtime": { "version": "1.1.6", "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.6.tgz", @@ -623,6 +1172,24 @@ "@emnapi/runtime": "^1.7.1" } }, + "node_modules/@opentelemetry/api": { + "version": "1.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.0.tgz", + "integrity": "sha512-3giAOQvZiH5F9bMlMiv8+GSPMeqg0dbaeo58/0SlA9sxSqZhnUtxzX9/2FzyhS9sWQf5S0GJE0AKBrFqjpeYcg==", + "license": "Apache-2.0", + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/@opentelemetry/semantic-conventions": { + "version": "1.41.1", + "resolved": "https://registry.npmjs.org/@opentelemetry/semantic-conventions/-/semantic-conventions-1.41.1.tgz", + "integrity": "sha512-/UhIkaZgPutTFmQ7RnIJGgDXZmtEJ7Dvi86xNTFWcnRxVRNk/aotsqDJYeEvDP+FSMB2SdW+pQzNMcWP0rwuNA==", + "license": "Apache-2.0", + "engines": { + "node": ">=14" + } + }, "node_modules/@oxc-project/types": { "version": "0.137.0", "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.137.0.tgz", @@ -633,6 +1200,63 @@ "url": "https://github.com/sponsors/Boshen" } }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz", + "integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz", + "integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz", + "integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==", + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.1.tgz", + "integrity": "sha512-oOAWABowe8EAbMyWKM0tYDKi8Yaox52D+HWZhAIJqQXbqe0xI/GV7FhLWqlEKreMkfDjshR5FKgi3mnle0h6Eg==", + "license": "BSD-3-Clause" + }, "node_modules/@quansync/fs": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/@quansync/fs/-/fs-1.0.0.tgz", @@ -1260,6 +1884,126 @@ "win32" ] }, + "node_modules/@smithy/core": { + "version": "3.26.0", + "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.26.0.tgz", + "integrity": "sha512-mLUktFAn+Pa2agl1J7VgtYNFWCX8/b4GMJSK1hCu4YCvtBfM6F8Os3EP4ry+DFFlXOf3wyvlgXhuUdFoy52D3g==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/crc32": "5.2.0", + "@smithy/types": "^4.15.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/credential-provider-imds": { + "version": "4.4.2", + "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.4.2.tgz", + "integrity": "sha512-18UMDMyrAbDcpmL1gLUA7ww0fRTcdCrSjSJOi2Sbld+tVjwD/pW+OAwjlScFLR7vvBnhZrIPQ7kVuTf1mnJLug==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.26.0", + "@smithy/types": "^4.15.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/fetch-http-handler": { + "version": "5.5.2", + "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.5.2.tgz", + "integrity": "sha512-Ei/UK/QMhq0rKaMqGPlOAkE2yS9DZeYmZdk1RAKc3vp3zxgleZHZyBLlZv8yLsxljX4svCRuMTD6u3LLIcU4Bg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.26.0", + "@smithy/types": "^4.15.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/is-array-buffer": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@smithy/is-array-buffer/-/is-array-buffer-2.2.0.tgz", + "integrity": "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@smithy/node-http-handler": { + "version": "4.7.3", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.7.3.tgz", + "integrity": "sha512-/jPhevcTFPMVl6KNjbaI47iOg1zxC7IsnX4PQDGVZKMFceOXtB8IEYaB7a9VvkP/3oC60WzTeKocvSI7vLT0vA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.24.3", + "@smithy/types": "^4.14.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/signature-v4": { + "version": "5.5.2", + "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.5.2.tgz", + "integrity": "sha512-7xHpmPY4rt0IOmeAA8EfjgEH8isT+587TCdy9H6a7d4OMi5CQ0oEHhWllunvPu4j4Cq0vTFwdxXN/kABWPjdyA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.26.0", + "@smithy/types": "^4.15.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/types": { + "version": "4.15.0", + "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.15.0.tgz", + "integrity": "sha512-Z5TAOxygoFvybJV3igo5SloFflSokHx2hu1eFA+DxDTcn+FtKxUSui+rbTRG1pAafMA888Z3MVvCWUuvCrTXjg==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-buffer-from": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@smithy/util-buffer-from/-/util-buffer-from-2.2.0.tgz", + "integrity": "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/is-array-buffer": "^2.2.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@smithy/util-utf8": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@smithy/util-utf8/-/util-utf8-2.3.0.tgz", + "integrity": "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/util-buffer-from": "^2.2.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/@tybys/wasm-util": { "version": "0.10.3", "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.3.tgz", @@ -1307,12 +2051,17 @@ "version": "22.20.0", "resolved": "https://registry.npmjs.org/@types/node/-/node-22.20.0.tgz", "integrity": "sha512-QWlFW2wf3nTjC13/DqRnBpR4ZO36VJH/JVBkA/vcnmbTBNQIlnObqyqZE1tUR7+Ni23Lda8R1BxMfbXRpCUx5g==", - "dev": true, "license": "MIT", "dependencies": { "undici-types": "~6.21.0" } }, + "node_modules/@types/retry": { + "version": "0.12.0", + "resolved": "https://registry.npmjs.org/@types/retry/-/retry-0.12.0.tgz", + "integrity": "sha512-wWKOClTTiizcZhXnPY4wikVAwmdYHp8q6DmC+EJUzAMsycb7HB32Kh9RN4+0gExjmPmZSAQjgURXIGATPegAvA==", + "license": "MIT" + }, "node_modules/@vitest/expect": { "version": "3.2.6", "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.6.tgz", @@ -1428,6 +2177,15 @@ "url": "https://opencollective.com/vitest" } }, + "node_modules/agent-base": { + "version": "7.1.4", + "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", + "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==", + "license": "MIT", + "engines": { + "node": ">= 14" + } + }, "node_modules/ansis": { "version": "4.3.1", "resolved": "https://registry.npmjs.org/ansis/-/ansis-4.3.1.tgz", @@ -1466,6 +2224,35 @@ "url": "https://github.com/sponsors/sxzz" } }, + "node_modules/base64-js": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", + "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/bignumber.js": { + "version": "9.3.1", + "resolved": "https://registry.npmjs.org/bignumber.js/-/bignumber.js-9.3.1.tgz", + "integrity": "sha512-Ko0uX15oIUS7wJ3Rb30Fs6SkVbLmPBAKdlm7q9+ak9bbIeFf0MwuBsQV6z7+X768/cHsfg+WlysDWJcmthjsjQ==", + "license": "MIT", + "engines": { + "node": "*" + } + }, "node_modules/birpc": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/birpc/-/birpc-4.0.0.tgz", @@ -1476,6 +2263,18 @@ "url": "https://github.com/sponsors/antfu" } }, + "node_modules/bowser": { + "version": "2.14.1", + "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.14.1.tgz", + "integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==", + "license": "MIT" + }, + "node_modules/buffer-equal-constant-time": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz", + "integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==", + "license": "BSD-3-Clause" + }, "node_modules/cac": { "version": "7.0.0", "resolved": "https://registry.npmjs.org/cac/-/cac-7.0.0.tgz", @@ -1513,11 +2312,19 @@ "node": ">= 16" } }, + "node_modules/data-uri-to-buffer": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-4.0.1.tgz", + "integrity": "sha512-0R9ikRb668HB7QDxT1vkpuUBtqc53YyAwMwGeUFKRojY/NWKvdZ+9UYtRfGmhqNbRkTSVpMbmyhXipFFv2cb/A==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, "node_modules/debug": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "dev": true, "license": "MIT", "dependencies": { "ms": "^2.1.3" @@ -1569,6 +2376,15 @@ } } }, + "node_modules/ecdsa-sig-formatter": { + "version": "1.0.11", + "resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz", + "integrity": "sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ==", + "license": "Apache-2.0", + "dependencies": { + "safe-buffer": "^5.0.1" + } + }, "node_modules/empathic": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/empathic/-/empathic-2.0.1.tgz", @@ -1648,6 +2464,12 @@ "node": ">=12.0.0" } }, + "node_modules/extend": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz", + "integrity": "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==", + "license": "MIT" + }, "node_modules/fdir": { "version": "6.5.0", "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", @@ -1666,6 +2488,41 @@ } } }, + "node_modules/fetch-blob": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/fetch-blob/-/fetch-blob-3.2.0.tgz", + "integrity": "sha512-7yAQpD2UMJzLi1Dqv7qFYnPbaPx7ZfFK6PiIxQ4PfkGPyNyl2Ugx+a/umUonmKqjhM4DnfbMvdX6otXq83soQQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/jimmywarting" + }, + { + "type": "paypal", + "url": "https://paypal.me/jimmywarting" + } + ], + "license": "MIT", + "dependencies": { + "node-domexception": "^1.0.0", + "web-streams-polyfill": "^3.0.3" + }, + "engines": { + "node": "^12.20 || >= 14.13" + } + }, + "node_modules/formdata-polyfill": { + "version": "4.0.10", + "resolved": "https://registry.npmjs.org/formdata-polyfill/-/formdata-polyfill-4.0.10.tgz", + "integrity": "sha512-buewHzMvYL29jdeQTVILecSaZKnt/RJWjoZCF5OW60Z67/GmSLBkOFM7qh1PI3zFNtJbaZL5eQu1vLfazOwj4g==", + "license": "MIT", + "dependencies": { + "fetch-blob": "^3.1.2" + }, + "engines": { + "node": ">=12.20.0" + } + }, "node_modules/fsevents": { "version": "2.3.3", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", @@ -1681,6 +2538,34 @@ "node": "^8.16.0 || ^10.6.0 || >=11.0.0" } }, + "node_modules/gaxios": { + "version": "7.1.5", + "resolved": "https://registry.npmjs.org/gaxios/-/gaxios-7.1.5.tgz", + "integrity": "sha512-5FZy72Rh8LhtjmvDrKkI+lVhrsQrVKVsItxMoDm5mNQE+xR0WVIIs+jzPSJgBvKVsLi24fZhXJIsNI0bihDzFg==", + "license": "Apache-2.0", + "dependencies": { + "extend": "^3.0.2", + "https-proxy-agent": "^7.0.1", + "node-fetch": "^3.3.2" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/gcp-metadata": { + "version": "8.1.2", + "resolved": "https://registry.npmjs.org/gcp-metadata/-/gcp-metadata-8.1.2.tgz", + "integrity": "sha512-zV/5HKTfCeKWnxG0Dmrw51hEWFGfcF2xiXqcA3+J90WDuP0SvoiSO5ORvcBsifmx/FoIjgQN3oNOGaQ5PhLFkg==", + "license": "Apache-2.0", + "dependencies": { + "gaxios": "^7.0.0", + "google-logging-utils": "^1.0.0", + "json-bigint": "^1.0.0" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/get-tsconfig": { "version": "5.0.0-beta.5", "resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-5.0.0-beta.5.tgz", @@ -1697,6 +2582,32 @@ "url": "https://github.com/privatenumber/get-tsconfig?sponsor=1" } }, + "node_modules/google-auth-library": { + "version": "10.9.0", + "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-10.9.0.tgz", + "integrity": "sha512-xtvUqvINPhTaBm7nXqlYPcrMHJPm1lCNdSovxnKKhTm+4JsvQ+KGVYJViLoH9Yxu8w+T0Qv5HubzYT9BLrppJg==", + "license": "Apache-2.0", + "dependencies": { + "base64-js": "^1.3.0", + "ecdsa-sig-formatter": "^1.0.11", + "gaxios": "^7.1.4", + "gcp-metadata": "8.1.2", + "google-logging-utils": "1.1.3", + "jws": "^4.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/google-logging-utils": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/google-logging-utils/-/google-logging-utils-1.1.3.tgz", + "integrity": "sha512-eAmLkjDjAFCVXg7A1unxHsLf961m6y17QFqXqAXGj/gVkKFrEICfStRfwUlGNfeCEjNRa32JEWOUTlYXPyyKvA==", + "license": "Apache-2.0", + "engines": { + "node": ">=14" + } + }, "node_modules/hookable": { "version": "6.1.1", "resolved": "https://registry.npmjs.org/hookable/-/hookable-6.1.1.tgz", @@ -1704,6 +2615,32 @@ "dev": true, "license": "MIT" }, + "node_modules/http-proxy-agent": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz", + "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.0", + "debug": "^4.3.4" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/https-proxy-agent": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz", + "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.2", + "debug": "4" + }, + "engines": { + "node": ">= 14" + } + }, "node_modules/import-without-cache": { "version": "0.4.0", "resolved": "https://registry.npmjs.org/import-without-cache/-/import-without-cache-0.4.0.tgz", @@ -1737,6 +2674,55 @@ "node": ">=6" } }, + "node_modules/json-bigint": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-bigint/-/json-bigint-1.0.0.tgz", + "integrity": "sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ==", + "license": "MIT", + "dependencies": { + "bignumber.js": "^9.0.0" + } + }, + "node_modules/json-schema-to-ts": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz", + "integrity": "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==", + "license": "MIT", + "dependencies": { + "@babel/runtime": "^7.18.3", + "ts-algebra": "^2.0.0" + }, + "engines": { + "node": ">=16" + } + }, + "node_modules/jwa": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz", + "integrity": "sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg==", + "license": "MIT", + "dependencies": { + "buffer-equal-constant-time": "^1.0.1", + "ecdsa-sig-formatter": "1.0.11", + "safe-buffer": "^5.0.1" + } + }, + "node_modules/jws": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/jws/-/jws-4.0.1.tgz", + "integrity": "sha512-EKI/M/yqPncGUUh44xz0PxSidXFr/+r0pA70+gIYhjv+et7yxM+s29Y+VGDkovRofQem0fs7Uvf4+YmAdyRduA==", + "license": "MIT", + "dependencies": { + "jwa": "^2.0.1", + "safe-buffer": "^5.0.1" + } + }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, "node_modules/loupe": { "version": "3.2.1", "resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz", @@ -1758,7 +2744,6 @@ "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "dev": true, "license": "MIT" }, "node_modules/nanoid": { @@ -1780,6 +2765,44 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, + "node_modules/node-domexception": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", + "integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==", + "deprecated": "Use your platform's native DOMException instead", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/jimmywarting" + }, + { + "type": "github", + "url": "https://paypal.me/jimmywarting" + } + ], + "license": "MIT", + "engines": { + "node": ">=10.5.0" + } + }, + "node_modules/node-fetch": { + "version": "3.3.2", + "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-3.3.2.tgz", + "integrity": "sha512-dRB78srN/l6gqWulah9SrxeYnxeddIG30+GOqK/9OlLVyLg3HPnr6SqOWTWOXKRwC2eGYCkZ59NNuSgvSrpgOA==", + "license": "MIT", + "dependencies": { + "data-uri-to-buffer": "^4.0.0", + "fetch-blob": "^3.1.4", + "formdata-polyfill": "^4.0.10" + }, + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/node-fetch" + } + }, "node_modules/obug": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/obug/-/obug-2.1.3.tgz", @@ -1794,6 +2817,46 @@ "node": ">=12.20.0" } }, + "node_modules/openai": { + "version": "6.26.0", + "resolved": "https://registry.npmjs.org/openai/-/openai-6.26.0.tgz", + "integrity": "sha512-zd23dbWTjiJ6sSAX6s0HrCZi41JwTA1bQVs0wLQPZ2/5o2gxOJA5wh7yOAUgwYybfhDXyhwlpeQf7Mlgx8EOCA==", + "license": "Apache-2.0", + "bin": { + "openai": "bin/cli" + }, + "peerDependencies": { + "ws": "^8.18.0", + "zod": "^3.25 || ^4.0" + }, + "peerDependenciesMeta": { + "ws": { + "optional": true + }, + "zod": { + "optional": true + } + } + }, + "node_modules/p-retry": { + "version": "4.6.2", + "resolved": "https://registry.npmjs.org/p-retry/-/p-retry-4.6.2.tgz", + "integrity": "sha512-312Id396EbJdvRONlngUx0NydfrIQ5lsYu0znKVUzVvArzEIt08V1qhtyESbGVd1FGX7UKtiFp5uwKZdM8wIuQ==", + "license": "MIT", + "dependencies": { + "@types/retry": "0.12.0", + "retry": "^0.13.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/partial-json": { + "version": "0.1.7", + "resolved": "https://registry.npmjs.org/partial-json/-/partial-json-0.1.7.tgz", + "integrity": "sha512-Njv/59hHaokb/hRUjce3Hdv12wd60MtM9Z5Olmn+nehe0QDAsRtRbJPvJ0Z91TusF0SuZRIvnM+S4l6EIP8leA==", + "license": "MIT" + }, "node_modules/pathe": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", @@ -1860,6 +2923,29 @@ "node": "^10 || ^12 || >=14" } }, + "node_modules/protobufjs": { + "version": "7.6.4", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.4.tgz", + "integrity": "sha512-RJJPTTpvFfHcWLkIa2JFWK4XvtSzS0yEWDmunqHXli1h3JlkbcQZXDZdcWxv+JK3Xsl5/UFDPZ0iGm7DAengYw==", + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.5", + "@protobufjs/eventemitter": "^1.1.1", + "@protobufjs/fetch": "^1.1.1", + "@protobufjs/float": "^1.0.2", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.1", + "@types/node": ">=13.7.0", + "long": "^5.3.2" + }, + "engines": { + "node": ">=12.0.0" + } + }, "node_modules/quansync": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/quansync/-/quansync-1.0.0.tgz", @@ -1887,6 +2973,15 @@ "url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1" } }, + "node_modules/retry": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/retry/-/retry-0.13.1.tgz", + "integrity": "sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg==", + "license": "MIT", + "engines": { + "node": ">= 4" + } + }, "node_modules/rolldown": { "version": "1.1.3", "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.1.3.tgz", @@ -2010,6 +3105,26 @@ "fsevents": "~2.3.2" } }, + "node_modules/safe-buffer": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", + "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, "node_modules/semver": { "version": "7.8.5", "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", @@ -2141,6 +3256,12 @@ "tree-kill": "cli.js" } }, + "node_modules/ts-algebra": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz", + "integrity": "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==", + "license": "MIT" + }, "node_modules/tsdown": { "version": "0.22.3", "resolved": "https://registry.npmjs.org/tsdown/-/tsdown-0.22.3.tgz", @@ -2218,9 +3339,13 @@ "version": "2.8.1", "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", - "dev": true, - "license": "0BSD", - "optional": true + "license": "0BSD" + }, + "node_modules/typebox": { + "version": "1.1.38", + "resolved": "https://registry.npmjs.org/typebox/-/typebox-1.1.38.tgz", + "integrity": "sha512-pZ0aQPmMmXoUvSbeuWf/Hzsc+avNw/Zd6VeE8CFgkVGWyuHPJvqeJJDeJqLve+K70LvjYIoleGcoJHPT17cWoA==", + "license": "MIT" }, "node_modules/typescript": { "version": "5.9.3", @@ -2254,7 +3379,6 @@ "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", - "dev": true, "license": "MIT" }, "node_modules/vite": { @@ -2445,6 +3569,15 @@ "dev": true, "license": "MIT" }, + "node_modules/web-streams-polyfill": { + "version": "3.3.3", + "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-3.3.3.tgz", + "integrity": "sha512-d2JWLCivmZYTSIoge9MsgFCZrt571BikcWGYkjC1khllbTeDlGqZ2D8vD8E/lJa8WGWbb7Plm8/XJYV7IJHZZw==", + "license": "MIT", + "engines": { + "node": ">= 8" + } + }, "node_modules/why-is-node-running": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", @@ -2461,6 +3594,45 @@ "engines": { "node": ">=8" } + }, + "node_modules/ws": { + "version": "8.21.0", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz", + "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==", + "license": "MIT", + "engines": { + "node": ">=10.0.0" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + } + }, + "node_modules/zod": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.4.3.tgz", + "integrity": "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + }, + "node_modules/zod-to-json-schema": { + "version": "3.25.2", + "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.2.tgz", + "integrity": "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==", + "license": "ISC", + "peerDependencies": { + "zod": "^3.25.28 || ^4" + } } } } diff --git a/benchmarks/adapters/online-mind2web/judge/package.json b/benchmarks/adapters/online-mind2web/judge/package.json index 2079044..0f3dd86 100644 --- a/benchmarks/adapters/online-mind2web/judge/package.json +++ b/benchmarks/adapters/online-mind2web/judge/package.json @@ -16,6 +16,9 @@ "engines": { "node": ">=22.0.0" }, + "dependencies": { + "@earendil-works/pi-ai": "^0.79.1" + }, "devDependencies": { "@types/node": "^22.10.0", "tsdown": "^0.22.2", diff --git a/benchmarks/adapters/online-mind2web/judge/src/model.ts b/benchmarks/adapters/online-mind2web/judge/src/model.ts index 13a59f6..045f13f 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/model.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/model.ts @@ -1,182 +1,67 @@ -import type { JudgeContent, JudgeModel } from "./types.ts"; +import { completeSimple, getEnvApiKey, getModel } from "@earendil-works/pi-ai"; +import type { Api, Model } from "@earendil-works/pi-ai"; +import type { JudgeModel } from "./types.ts"; + +// `getModel`/`getEnvApiKey` are typed against the literal provider/model +// registry, but the judge resolves them from a runtime `provider:name` ref. +// Reach for the runtime signatures: both look up the registry and return +// undefined for an unknown provider or model. +const resolveModel = getModel as (provider: string, modelId: string) => Model | undefined; +const resolveApiKey = getEnvApiKey as (provider: string) => string | undefined; /** - * {@link JudgeModel} implementations backed by raw `fetch` calls to a provider's - * chat API. Self-contained so the bundled judge runs inside the Kernel browser - * VM with no `npm install` at verify time (the VM ships `node` and global - * `fetch` but not the cua SDKs or any provider SDK). + * {@link JudgeModel} backed by `@earendil-works/pi-ai`. pi-ai owns provider + * routing, env-key resolution (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …), the + * o-series reasoning quirks (no `temperature`, `max_completion_tokens`), and + * vision encoding, so the judge no longer hand-rolls per-provider chat clients. + * pi-ai is bundled into the verifier (tsdown, no externals) so it stays + * self-contained inside the Kernel browser VM with no `npm install` at verify + * time. * * The judge model is given as a `provider:name` ref. `openai` is the default * provider — canonical WebJudge runs on an OpenAI o-series backbone (o4-mini), - * which is what the published leaderboard numbers are calibrated to. `anthropic` - * stays wired as a configurable, non-canonical alternative. + * which is what the published leaderboard numbers are calibrated to. */ -const OPENAI_URL = "https://api.openai.com/v1/chat/completions"; -const ANTHROPIC_URL = "https://api.anthropic.com/v1/messages"; -const ANTHROPIC_VERSION = "2023-06-01"; // Upstream caps each stage at 512 (max_new_tokens). We keep 1024 of headroom so // the verbose judge's per-image reasoning isn't truncated before the // `Score`/`Status:` line the parsers key on. const MAX_OUTPUT_TOKENS = 1024; -interface AnthropicBlock { - type: string; - text?: string; -} - export function parseModelRef(ref: string): { provider: string; name: string } { const idx = ref.indexOf(":"); if (idx === -1) return { provider: "openai", name: ref }; return { provider: ref.slice(0, idx), name: ref.slice(idx + 1) }; } -/** Provider dispatch for `JUDGE_MODEL` refs. Defaults to OpenAI when no prefix. */ +/** Resolve a `JUDGE_MODEL` ref to a pi-ai-backed {@link JudgeModel}. Defaults to OpenAI when no prefix. */ export function judgeModel(ref: string): JudgeModel { - const { provider } = parseModelRef(ref); - switch (provider) { - case "openai": - return openaiJudgeModel(ref); - case "anthropic": - return anthropicJudgeModel(ref); - default: - throw new Error( - `unsupported judge provider "${provider}" (openai or anthropic); got "${ref}"`, - ); - } -} - -/** o-series reasoning models (o4-mini, o3, …) reject `temperature` and need - * `max_completion_tokens` instead of `max_tokens`. */ -function isOSeries(name: string): boolean { - return /^o\d/.test(name); -} - -function toOpenAIContent(content: JudgeContent): unknown[] { - return content.map((part) => - part.type === "text" - ? { type: "text", text: part.text } - : { - type: "image_url", - // WebJudge sends `detail: high` so the judge scores the full screenshot. - image_url: { - url: `data:${part.mimeType};base64,${part.data}`, - detail: "high", - }, - }, - ); -} - -/** OpenAI Chat Completions judge. Resolves the API key from `OPENAI_API_KEY`. */ -export function openaiJudgeModel(ref: string): JudgeModel { const { provider, name } = parseModelRef(ref); - if (provider !== "openai") { - throw new Error(`openaiJudgeModel got a non-openai ref "${ref}"`); + const model = resolveModel(provider, name); + if (!model) { + throw new Error(`unknown judge model "${ref}"`); } - const apiKey = process.env.OPENAI_API_KEY; + const apiKey = resolveApiKey(provider); if (!apiKey) { - throw new Error("OPENAI_API_KEY is required for the WebJudge model"); + throw new Error(`no API key in the environment for judge provider "${provider}"`); } - const oSeries = isOSeries(name); - const headers = { - "content-type": "application/json", - authorization: `Bearer ${apiKey}`, - }; return { async complete(systemPrompt, content) { - const body: Record = { - model: name, - // o-series uses max_completion_tokens; older chat models use max_tokens. - [oSeries ? "max_completion_tokens" : "max_tokens"]: MAX_OUTPUT_TOKENS, - messages: [ - { role: "system", content: systemPrompt }, - { role: "user", content: toOpenAIContent(content) }, - ], - }; - // WebJudge wants deterministic scoring (temperature 0), but o-series - // reasoning models reject the field outright — omit it for o-series. - if (!oSeries) body.temperature = 0; - const res = await fetch(OPENAI_URL, { - method: "POST", - headers, - body: JSON.stringify(body), - }); - if (!res.ok) { - const errBody = await res.text(); - throw new Error(`OpenAI API error ${res.status}: ${errBody}`); - } - const data = (await res.json()) as { - choices?: Array<{ message?: { content?: string } }>; - }; - return data.choices?.[0]?.message?.content ?? ""; - }, - }; -} - -function toAnthropicContent(content: JudgeContent): unknown[] { - return content.map((part) => - part.type === "text" - ? { type: "text", text: part.text } - : { - type: "image", - source: { type: "base64", media_type: part.mimeType, data: part.data }, - }, - ); -} - -/** Anthropic Messages judge. Resolves the API key from `ANTHROPIC_API_KEY`. */ -export function anthropicJudgeModel(ref: string): JudgeModel { - const { provider, name } = parseModelRef(ref); - if (provider !== "anthropic") { - throw new Error(`anthropicJudgeModel got a non-anthropic ref "${ref}"`); - } - const apiKey = process.env.ANTHROPIC_API_KEY; - if (!apiKey) { - throw new Error("ANTHROPIC_API_KEY is required for the WebJudge model"); - } - const headers = { - "content-type": "application/json", - "x-api-key": apiKey, - "anthropic-version": ANTHROPIC_VERSION, - }; - - async function callOnce( - systemPrompt: string, - content: JudgeContent, - withTemperature: boolean, - ): Promise { - const body: Record = { - model: name, - max_tokens: MAX_OUTPUT_TOKENS, - system: systemPrompt, - messages: [{ role: "user", content: toAnthropicContent(content) }], - }; - // WebJudge wants deterministic scoring, but newer models (e.g. opus-4-8) - // reject `temperature` outright; on that 400 we retry without it. - if (withTemperature) body.temperature = 0; - return fetch(ANTHROPIC_URL, { method: "POST", headers, body: JSON.stringify(body) }); - } - - return { - async complete(systemPrompt, content) { - let res = await callOnce(systemPrompt, content, true); - if (res.status === 400) { - const body = await res.text(); - if (body.includes("temperature")) { - res = await callOnce(systemPrompt, content, false); - } else { - throw new Error(`Anthropic API error 400: ${body}`); - } - } - if (!res.ok) { - const body = await res.text(); - throw new Error(`Anthropic API error ${res.status}: ${body}`); + const res = await completeSimple( + model, + { systemPrompt, messages: [{ role: "user", content, timestamp: Date.now() }] }, + // WebJudge wants deterministic scoring (temperature 0); pi-ai drops the + // field for backbones that reject it. + { apiKey, temperature: 0, maxTokens: MAX_OUTPUT_TOKENS }, + ); + // pi-ai surfaces provider errors as a message with stopReason "error" + // rather than throwing. Raise it so the grading failure is logged to the + // verifier and never scored as a legitimate task failure. + if (res.stopReason === "error") { + throw new Error(`judge model error: ${res.errorMessage ?? "unknown error"}`); } - const data = (await res.json()) as { content?: AnthropicBlock[] }; - return (data.content ?? []) - .flatMap((b) => (b.type === "text" && b.text ? [b.text] : [])) - .join(""); + return res.content.flatMap((c) => (c.type === "text" ? [c.text] : [])).join(""); }, }; } diff --git a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts index 29047d7..20deedd 100644 --- a/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts +++ b/benchmarks/adapters/online-mind2web/judge/test/artifacts.test.ts @@ -4,14 +4,29 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; import { loadTask, loadTrajectory } from "../src/artifacts.ts"; -import { - anthropicJudgeModel, - judgeModel, - openaiJudgeModel, - parseModelRef, -} from "../src/model.ts"; import type { JudgeContent } from "../src/types.ts"; +// Stub pi-ai so the judge tests stay offline: `getModel`/`getEnvApiKey` own the +// provider+key resolution and `completeSimple` owns the network call, all of +// which pi-ai covers and the judge just wires together. +const { completeSimple, getModel, getEnvApiKey } = vi.hoisted(() => ({ + completeSimple: vi.fn(), + getModel: vi.fn(), + getEnvApiKey: vi.fn(), +})); +vi.mock("@earendil-works/pi-ai", () => ({ completeSimple, getModel, getEnvApiKey })); + +const { judgeModel, parseModelRef } = await import("../src/model.ts"); + +/** A pi-ai AssistantMessage carrying a single text block. */ +function assistantText(text: string) { + return { + role: "assistant" as const, + content: [{ type: "text" as const, text }], + stopReason: "stop" as const, + }; +} + // 1x1 transparent PNG, the same fixture the shared-core sink test uses. const PNG_B64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="; @@ -137,174 +152,102 @@ describe("parseModelRef", () => { }); }); -describe("anthropicJudgeModel", () => { +describe("judgeModel", () => { afterEach(() => { - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); - - function okResponse(text: string): Response { - return new Response(JSON.stringify({ content: [{ type: "text", text }] }), { status: 200 }); - } - - it("retries without temperature when the model rejects it (400)", async () => { - vi.stubEnv("ANTHROPIC_API_KEY", "k"); - const bodies: Array> = []; - const fetchMock = vi.fn(async (_url: string, init: RequestInit) => { - bodies.push(JSON.parse(init.body as string)); - if (bodies.length === 1) { - return new Response( - JSON.stringify({ error: { message: "`temperature` is deprecated for this model." } }), - { status: 400 }, - ); - } - return okResponse("verdict"); - }); - vi.stubGlobal("fetch", fetchMock); - - const out = await anthropicJudgeModel("anthropic:claude-opus-4-8").complete("sys", [ - { type: "text", text: "hi" }, - ]); - - expect(out).toBe("verdict"); - expect(fetchMock).toHaveBeenCalledTimes(2); - expect(bodies[0].temperature).toBe(0); - expect(bodies[1]).not.toHaveProperty("temperature"); + completeSimple.mockReset(); + getModel.mockReset(); + getEnvApiKey.mockReset(); }); - it("throws on a 400 unrelated to temperature", async () => { - vi.stubEnv("ANTHROPIC_API_KEY", "k"); - vi.stubGlobal( - "fetch", - vi.fn(async () => new Response(JSON.stringify({ error: { message: "bad request" } }), { status: 400 })), - ); - - await expect( - anthropicJudgeModel("anthropic:claude-opus-4-8").complete("sys", [{ type: "text", text: "hi" }]), - ).rejects.toThrow("Anthropic API error 400"); - }); -}); - -describe("openaiJudgeModel", () => { - afterEach(() => { - vi.unstubAllGlobals(); - vi.unstubAllEnvs(); - }); - - function okResponse(text: string): Response { - return new Response(JSON.stringify({ choices: [{ message: { content: text } }] }), { - status: 200, - }); - } - - /** Stub fetch with an OK response and capture each request body. */ - function captureBodies(text = "out"): Array> { - const bodies: Array> = []; - vi.stubGlobal( - "fetch", - vi.fn(async (_url: string, init: RequestInit) => { - bodies.push(JSON.parse(init.body as string)); - return okResponse(text); - }), - ); - return bodies; - } - const vision: JudgeContent = [ { type: "text", text: "score this" }, { type: "image", data: "AAAA", mimeType: "image/png" }, ]; - it("omits temperature and uses max_completion_tokens for o-series", async () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - const bodies = captureBodies(); + it("resolves the ref through pi-ai and passes the prompt, content, and deterministic options", async () => { + const model = { id: "o4-mini", provider: "openai" }; + getModel.mockReturnValue(model); + getEnvApiKey.mockReturnValue("sk-test"); + completeSimple.mockResolvedValue(assistantText("Status: success")); - const out = await openaiJudgeModel("openai:o4-mini").complete("sys", [ - { type: "text", text: "hi" }, - ]); + const out = await judgeModel("openai:o4-mini").complete("sys", vision); - expect(out).toBe("out"); - expect(bodies[0]).not.toHaveProperty("temperature"); - expect(bodies[0].max_completion_tokens).toBe(1024); - expect(bodies[0]).not.toHaveProperty("max_tokens"); - expect(bodies[0].model).toBe("o4-mini"); - }); - - it("sends temperature 0 and max_tokens for non-o-series chat models", async () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - const bodies = captureBodies(); - - await openaiJudgeModel("openai:gpt-4o").complete("sys", [{ type: "text", text: "hi" }]); + expect(out).toBe("Status: success"); + expect(getModel).toHaveBeenCalledWith("openai", "o4-mini"); + expect(getEnvApiKey).toHaveBeenCalledWith("openai"); - expect(bodies[0].temperature).toBe(0); - expect(bodies[0].max_tokens).toBe(1024); - expect(bodies[0]).not.toHaveProperty("max_completion_tokens"); + const [calledModel, context, options] = completeSimple.mock.calls[0]; + expect(calledModel).toBe(model); + expect(context.systemPrompt).toBe("sys"); + expect(context.messages[0].role).toBe("user"); + expect(context.messages[0].content).toBe(vision); + expect(options).toMatchObject({ apiKey: "sk-test", temperature: 0, maxTokens: 1024 }); }); - it("encodes images as data-url image_url with detail:high", async () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - const bodies = captureBodies(); - - await openaiJudgeModel("openai:o4-mini").complete("sys", vision); - - const messages = bodies[0].messages as Array<{ role: string; content: unknown }>; - expect(messages[0]).toEqual({ role: "system", content: "sys" }); - expect(messages[1].content).toEqual([ - { type: "text", text: "score this" }, - { - type: "image_url", - image_url: { url: "data:image/png;base64,AAAA", detail: "high" }, - }, - ]); - }); + it("defaults a bare ref to the openai provider", async () => { + getModel.mockReturnValue({ id: "o4-mini" }); + getEnvApiKey.mockReturnValue("sk-test"); + completeSimple.mockResolvedValue(assistantText("ok")); - it("reads the assistant content from the first choice", async () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - captureBodies("Status: success"); + await judgeModel("o4-mini").complete("sys", [{ type: "text", text: "hi" }]); - const out = await openaiJudgeModel("openai:o4-mini").complete("sys", vision); - expect(out).toBe("Status: success"); + expect(getModel).toHaveBeenCalledWith("openai", "o4-mini"); }); - it("throws when the API key is missing", () => { - vi.stubEnv("OPENAI_API_KEY", ""); - expect(() => openaiJudgeModel("openai:o4-mini")).toThrow("OPENAI_API_KEY is required"); - }); + it("routes an anthropic ref to the anthropic provider", async () => { + getModel.mockReturnValue({ id: "claude-opus-4-8" }); + getEnvApiKey.mockReturnValue("sk-ant"); + completeSimple.mockResolvedValue(assistantText("ok")); - it("throws on a non-2xx response", async () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - vi.stubGlobal( - "fetch", - vi.fn(async () => new Response(JSON.stringify({ error: { message: "rate limit" } }), { status: 429 })), - ); + await judgeModel("anthropic:claude-opus-4-8").complete("sys", [{ type: "text", text: "hi" }]); - await expect( - openaiJudgeModel("openai:o4-mini").complete("sys", [{ type: "text", text: "hi" }]), - ).rejects.toThrow("OpenAI API error 429"); + expect(getModel).toHaveBeenCalledWith("anthropic", "claude-opus-4-8"); + expect(getEnvApiKey).toHaveBeenCalledWith("anthropic"); + expect(completeSimple.mock.calls[0][2]).toMatchObject({ apiKey: "sk-ant" }); }); -}); -describe("judgeModel routing", () => { - afterEach(() => { - vi.unstubAllEnvs(); - }); + it("concatenates the text blocks of the assistant message", async () => { + getModel.mockReturnValue({ id: "o4-mini" }); + getEnvApiKey.mockReturnValue("sk-test"); + completeSimple.mockResolvedValue({ + role: "assistant", + content: [ + { type: "thinking", thinking: "reasoning..." }, + { type: "text", text: "Status: " }, + { type: "text", text: "success" }, + ], + stopReason: "stop", + }); - it("routes openai: refs to the OpenAI client", () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - expect(() => judgeModel("openai:o4-mini")).not.toThrow(); + const out = await judgeModel("openai:o4-mini").complete("sys", vision); + expect(out).toBe("Status: success"); }); - it("routes a bare ref to OpenAI (default provider)", () => { - vi.stubEnv("OPENAI_API_KEY", "k"); - expect(() => judgeModel("o4-mini")).not.toThrow(); + it("throws when pi-ai returns a provider error instead of throwing", async () => { + getModel.mockReturnValue({ id: "o4-mini" }); + getEnvApiKey.mockReturnValue("sk-test"); + completeSimple.mockResolvedValue({ + role: "assistant", + content: [], + stopReason: "error", + errorMessage: "OpenAI API error (401): Incorrect API key", + }); + + await expect(judgeModel("openai:o4-mini").complete("sys", vision)).rejects.toThrow( + "judge model error: OpenAI API error (401): Incorrect API key", + ); }); - it("routes anthropic: refs to the Anthropic client", () => { - vi.stubEnv("ANTHROPIC_API_KEY", "k"); - expect(() => judgeModel("anthropic:claude-opus-4-8")).not.toThrow(); + it("throws when the model ref is unknown to pi-ai", () => { + getModel.mockReturnValue(undefined); + expect(() => judgeModel("gemini:flash")).toThrow('unknown judge model "gemini:flash"'); }); - it("throws on an unsupported provider", () => { - expect(() => judgeModel("gemini:flash")).toThrow('unsupported judge provider "gemini"'); + it("throws when no API key is configured for the provider", () => { + getModel.mockReturnValue({ id: "o4-mini" }); + getEnvApiKey.mockReturnValue(undefined); + expect(() => judgeModel("openai:o4-mini")).toThrow( + 'no API key in the environment for judge provider "openai"', + ); }); }); diff --git a/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts b/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts index ad32eab..6733852 100644 --- a/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts +++ b/benchmarks/adapters/online-mind2web/judge/tsdown.config.ts @@ -1,8 +1,9 @@ import { defineConfig } from "tsdown"; // Bundle to a single self-contained ESM file so `node dist/judge.js` runs inside -// the Kernel VM with no dependency install. No externals: the judge only uses -// node builtins and global fetch. +// the Kernel VM with no dependency install. pi-ai is bundled in (noExternal); it +// lazy-loads providers via dynamic import(), so inlineDynamicImports folds those +// chunks back into the one judge.js the adapter copies next to the verifier. export default defineConfig({ entry: ["src/judge.ts"], format: ["esm"], @@ -11,5 +12,6 @@ export default defineConfig({ sourcemap: false, clean: true, noExternal: [/.*/], + outputOptions: { inlineDynamicImports: true }, outExtensions: () => ({ js: ".js" }), }); From b3816424ebd9a9fc648e2d2c601fbd64ad5e6909 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 22:08:21 +0000 Subject: [PATCH 6/9] online-mind2web: send o-series reasoning effort, drop temperature in the pi-ai judge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit o4-mini (and o3) reject `temperature` and reject a reasoning effort of `none` — pi-ai's default when the level is unset — on the responses API. Pass `reasoning: medium` (o4-mini's own default, the calibration the published WebJudge numbers use) and omit `temperature` for OpenAI reasoning models; every other backbone keeps deterministic `temperature: 0`. Verified live: o4-mini grades a real trajectory end-to-end (key points, per-screenshot vision scores, verdict) with no API errors. Co-Authored-By: Claude Opus 4.7 --- .../adapters/online-mind2web/judge/src/model.ts | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/benchmarks/adapters/online-mind2web/judge/src/model.ts b/benchmarks/adapters/online-mind2web/judge/src/model.ts index 045f13f..39a5000 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/model.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/model.ts @@ -46,14 +46,24 @@ export function judgeModel(ref: string): JudgeModel { throw new Error(`no API key in the environment for judge provider "${provider}"`); } + // OpenAI reasoning backbones (o4-mini, o3, …) don't accept `temperature` and + // reject a reasoning effort of "none" (pi-ai's default when the level is + // unset) — they require low/medium/high. "medium" is OpenAI's own o4-mini + // default, the setting the published WebJudge agreement numbers are + // calibrated to. Every other backbone keeps deterministic scoring + // (temperature 0) with no forced thinking. + const baseOptions = { apiKey, maxTokens: MAX_OUTPUT_TOKENS }; + const options = + model.reasoning && provider === "openai" + ? { ...baseOptions, reasoning: "medium" as const } + : { ...baseOptions, temperature: 0 }; + return { async complete(systemPrompt, content) { const res = await completeSimple( model, { systemPrompt, messages: [{ role: "user", content, timestamp: Date.now() }] }, - // WebJudge wants deterministic scoring (temperature 0); pi-ai drops the - // field for backbones that reject it. - { apiKey, temperature: 0, maxTokens: MAX_OUTPUT_TOKENS }, + options, ); // pi-ai surfaces provider errors as a message with stopReason "error" // rather than throwing. Raise it so the grading failure is logged to the From e739410f83e22c95cfc410ddb8e58a2059102769 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sat, 27 Jun 2026 22:50:29 +0000 Subject: [PATCH 7/9] online-mind2web: assert openai:o4-mini default in adapter test The judge backbone now defaults to openai:o4-mini (leaderboard parity), but test_generates_full_task_dir still asserted JUDGE_MODEL started with "anthropic:", so it failed against the current task.toml template. Co-Authored-By: Claude Opus 4.7 --- benchmarks/adapters/online-mind2web/tests/test_adapter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/adapters/online-mind2web/tests/test_adapter.py b/benchmarks/adapters/online-mind2web/tests/test_adapter.py index e05021e..e972160 100644 --- a/benchmarks/adapters/online-mind2web/tests/test_adapter.py +++ b/benchmarks/adapters/online-mind2web/tests/test_adapter.py @@ -112,7 +112,7 @@ def test_generates_full_task_dir(tmp_path): assert toml["task"]["name"] == "osunlp/online-mind2web-abc123-110325" assert toml["metadata"]["reference_length"] == "6" assert toml["metadata"]["difficulty"] == "medium" - assert toml["verifier"]["env"]["JUDGE_MODEL"].startswith("anthropic:") + assert toml["verifier"]["env"]["JUDGE_MODEL"] == "openai:o4-mini" instruction = (task_dir / "instruction.md").read_text() assert "https://www.traderjoes.com/" in instruction From 2f91eb546e81001760f692719a24c0a4d570b5de Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sun, 28 Jun 2026 13:09:24 +0000 Subject: [PATCH 8/9] online-mind2web: harden judge reward write and threshold parse MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two verifier-correctness fixes in the WebJudge entrypoint: - The grading_details.json write ran inside the same try as the reward write, so a details-artifact failure hit the shared catch and rewrote the reward to 0 — flipping a decided pass to fail. Make the reward the authoritative write and grade the details artifact best-effort so it can never overwrite a decided reward. - A non-numeric --score-threshold parsed to NaN, making every `score >= threshold` comparison false so no snapshots were graded. Fall back to the documented default of 3 for a non-finite value. Add unit tests covering both, gating main() behind an entrypoint check so the module is importable from the tests. Co-Authored-By: Claude Opus 4.7 --- .../online-mind2web/judge/src/judge.ts | 54 ++++++++++---- .../online-mind2web/judge/test/judge.test.ts | 74 +++++++++++++++++++ 2 files changed, 112 insertions(+), 16 deletions(-) create mode 100644 benchmarks/adapters/online-mind2web/judge/test/judge.test.ts diff --git a/benchmarks/adapters/online-mind2web/judge/src/judge.ts b/benchmarks/adapters/online-mind2web/judge/src/judge.ts index 05f246b..3b71520 100644 --- a/benchmarks/adapters/online-mind2web/judge/src/judge.ts +++ b/benchmarks/adapters/online-mind2web/judge/src/judge.ts @@ -13,8 +13,11 @@ */ import { mkdirSync, writeFileSync } from "node:fs"; import { dirname } from "node:path"; +import { argv } from "node:process"; +import { fileURLToPath } from "node:url"; import { loadTask, loadTrajectory } from "./artifacts.ts"; import { judgeModel } from "./model.ts"; +import type { GradeResult } from "./types.ts"; import { gradeWithWebJudge } from "./webjudge.ts"; interface Args { @@ -28,7 +31,7 @@ interface Args { detailsOut?: string; } -function parseArgs(argv: string[]): Args { +export function parseArgs(argv: string[]): Args { const flags = new Map(); for (let i = 0; i < argv.length; i += 1) { const arg = argv[i]; @@ -42,13 +45,16 @@ function parseArgs(argv: string[]): Args { if (!value) throw new Error(`missing required --${name}`); return value; }; + // A non-numeric --score-threshold (NaN) would fail every `score >= threshold` + // comparison and silently grade zero snapshots, so fall back to the default 3. + const threshold = Number(flags.get("score-threshold") ?? "3"); return { task: require("task"), run: require("run"), answer: require("answer"), shots: flags.get("shots"), judgeModel: flags.get("judge-model") ?? "openai:o4-mini", - scoreThreshold: Number(flags.get("score-threshold") ?? "3"), + scoreThreshold: Number.isFinite(threshold) ? threshold : 3, rewardOut: require("reward-out"), detailsOut: flags.get("details-out"), }; @@ -59,8 +65,31 @@ function writeReward(path: string, reward: 0 | 1): void { writeFileSync(path, `${reward}\n`); } +/** + * Write the grading outcome. The reward is the authoritative output and is + * written first; the `grading_details.json` artifact is best-effort, so a + * failure writing it logs to stderr but never flips the decided reward. + */ +export function writeOutcome(args: Args, result: GradeResult): void { + writeReward(args.rewardOut, result.success ? 1 : 0); + if (!args.detailsOut) return; + try { + mkdirSync(dirname(args.detailsOut), { recursive: true }); + writeFileSync( + args.detailsOut, + JSON.stringify( + { success: result.success, reasoning: result.reasoning, details: result.details }, + null, + 2, + ), + ); + } catch (err) { + console.error(err instanceof Error ? (err.stack ?? err.message) : String(err)); + } +} + async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); + const args = parseArgs(argv.slice(2)); try { const task = loadTask(args.task); const trajectory = loadTrajectory({ @@ -75,18 +104,7 @@ async function main(): Promise { judge, scoreThreshold: args.scoreThreshold, }); - writeReward(args.rewardOut, result.success ? 1 : 0); - if (args.detailsOut) { - mkdirSync(dirname(args.detailsOut), { recursive: true }); - writeFileSync( - args.detailsOut, - JSON.stringify( - { success: result.success, reasoning: result.reasoning, details: result.details }, - null, - 2, - ), - ); - } + writeOutcome(args, result); } catch (err) { // Never leave the reward file empty: a grading failure is a 0, and the // error is surfaced on stderr for the verifier log. @@ -95,4 +113,8 @@ async function main(): Promise { } } -main(); +// Run only as the CLI entrypoint, so importing this module from a test does not +// fire grading against vitest's argv. +if (argv[1] && fileURLToPath(import.meta.url) === argv[1]) { + main(); +} diff --git a/benchmarks/adapters/online-mind2web/judge/test/judge.test.ts b/benchmarks/adapters/online-mind2web/judge/test/judge.test.ts new file mode 100644 index 0000000..ef8c16d --- /dev/null +++ b/benchmarks/adapters/online-mind2web/judge/test/judge.test.ts @@ -0,0 +1,74 @@ +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { parseArgs, writeOutcome } from "../src/judge.ts"; +import type { GradeResult } from "../src/types.ts"; + +const BASE = [ + "--task", + "/t/task.json", + "--run", + "/t/run.jsonl", + "--answer", + "/t/answer.txt", + "--reward-out", + "/t/reward.txt", +]; + +describe("parseArgs score-threshold", () => { + it("parses a numeric threshold", () => { + expect(parseArgs([...BASE, "--score-threshold", "4"]).scoreThreshold).toBe(4); + }); + + it("defaults to 3 when the flag is absent", () => { + expect(parseArgs(BASE).scoreThreshold).toBe(3); + }); + + it("falls back to 3 for a non-numeric threshold instead of NaN", () => { + expect(parseArgs([...BASE, "--score-threshold", "three"]).scoreThreshold).toBe(3); + }); + + it("keeps an explicit 0 threshold", () => { + expect(parseArgs([...BASE, "--score-threshold", "0"]).scoreThreshold).toBe(0); + }); +}); + +describe("writeOutcome", () => { + afterEach(() => vi.restoreAllMocks()); + + const pass: GradeResult = { success: true, reasoning: "Status: success", details: {} }; + + it("writes the decided reward", () => { + const dir = mkdtempSync(join(tmpdir(), "om2w-outcome-")); + const rewardOut = join(dir, "reward.txt"); + writeOutcome({ ...argsFor(rewardOut), detailsOut: join(dir, "details.json") }, pass); + expect(readFileSync(rewardOut, "utf8")).toBe("1\n"); + expect(JSON.parse(readFileSync(join(dir, "details.json"), "utf8")).success).toBe(true); + }); + + it("keeps the reward when the details write fails", () => { + const dir = mkdtempSync(join(tmpdir(), "om2w-outcome-")); + const rewardOut = join(dir, "reward.txt"); + // Make the details dir un-creatable: its parent is a regular file, so + // mkdirSync throws ENOTDIR. The reward must survive. + const fileAsDir = join(dir, "afile"); + writeFileSync(fileAsDir, ""); + vi.spyOn(console, "error").mockImplementation(() => {}); + + writeOutcome({ ...argsFor(rewardOut), detailsOut: join(fileAsDir, "details.json") }, pass); + + expect(readFileSync(rewardOut, "utf8")).toBe("1\n"); + }); +}); + +function argsFor(rewardOut: string) { + return { + task: "/t/task.json", + run: "/t/run.jsonl", + answer: "/t/answer.txt", + judgeModel: "openai:o4-mini", + scoreThreshold: 3, + rewardOut, + }; +} From 661d878719c3e2a0ee180deaee1acad8446d07b0 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Sun, 28 Jun 2026 15:30:10 +0000 Subject: [PATCH 9/9] online-mind2web: correct the judge-backbone README to the pi-ai transport The judge moved to @earendil-works/pi-ai, but the README still described the old hand-rolled client (omit/swap max_completion_tokens, image_url detail:high). Describe the current path: pi-ai owns provider routing, env-key resolution, the o-series quirks (medium reasoning effort + dropping temperature), and vision encoding; it is bundled into the verifier so the in-VM judge stays self-contained. Co-Authored-By: Claude Opus 4.7 --- benchmarks/adapters/online-mind2web/README.md | 21 +++++++++++-------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/benchmarks/adapters/online-mind2web/README.md b/benchmarks/adapters/online-mind2web/README.md index 69df537..2fcaca5 100644 --- a/benchmarks/adapters/online-mind2web/README.md +++ b/benchmarks/adapters/online-mind2web/README.md @@ -50,13 +50,16 @@ threshold are set in `[verifier.env]` (`JUDGE_MODEL`, `SCORE_THRESHOLD`). The default `JUDGE_MODEL` is **`openai:o4-mini`** — the published WebJudge backbone (~85.7% human agreement), so a recomputed success rate is comparable to -the Online-Mind2Web leaderboard. The judge bin parses the provider from the -`JUDGE_MODEL` prefix and reads the matching key from the verifier env: - -- `openai:` → `OPENAI_API_KEY`. o-series models (o4-mini, o3, …) reject - `temperature` and use `max_completion_tokens`; the client omits/swaps these - automatically. Screenshots are sent as vision `image_url` blocks with - `detail: high`. +the Online-Mind2Web leaderboard. The judge resolves the `JUDGE_MODEL` ref +(`provider:name`) through [`@earendil-works/pi-ai`](https://www.npmjs.com/package/@earendil-works/pi-ai), +which owns provider routing, env-key resolution, the OpenAI o-series quirks, and +vision encoding — so the bin carries no hand-rolled provider client. pi-ai is +bundled into the verifier (tsdown, no externals), so it stays self-contained in +the Kernel VM with no `npm install` at grade time. + +- `openai:` → `OPENAI_API_KEY`. For o-series reasoning models (o4-mini, + o3, …) the judge requests a `medium` reasoning effort and omits `temperature` + (which they reject); pi-ai handles the token cap and the vision encoding. - `anthropic:` → `ANTHROPIC_API_KEY`. Configurable, **non-canonical** cheaper alternative (e.g. `anthropic:claude-sonnet-4-6` / `anthropic:claude-opus-4-8`); an opus/sonnet judge is a *different grader* and @@ -66,8 +69,8 @@ Both keys are passed through `[verifier.env]`, resolved from host env, so either provider works by changing only `JUDGE_MODEL`. [WebJudge-7B](https://huggingface.co/osunlp/WebJudge-7B) (open weights) is a -future cheaper option but needs GPU hosting, so it is not wired into the -dependency-free in-VM bundle. +future cheaper option but needs GPU hosting, so it is not wired into the bundled +in-VM judge. ## Run on Harbor