Feature request
Add pi-coding-agent as a supported agent harness for @vercel/agent-eval.
agent-eval already supports several coding-agent CLIs through sandboxed command execution. PI support would let users compare PI against the same fixture, result artifact, and Playground workflow used for Claude Code, Codex, Gemini, Cursor, and OpenCode.
Suggested shape
- Add a built-in agent such as
pi-coding-agent or pi-cli.
- Prefer the same sandboxed command boundary used by existing agents: run PI as a CLI command or as a small isolated runner inside the sandbox, then capture stdout/stderr, transcript evidence, generated files, and exit status.
- Avoid importing a long-running agent SDK directly into the evaluator parent process if a CLI/child-process boundary can provide the same behavior. Child failures should become agent-run failures, not evaluator-process crashes.
- Support model/provider configuration through
agentOptions, including OpenAI-compatible or Azure-style endpoint settings if the PI runtime requires them.
- Preserve the existing result bundle and Playground compatibility so PI runs can be inspected and compared with other agents.
Acceptance criteria
- A config can select the PI harness, for example
agent: "pi-coding-agent" or agent: "pi-cli".
- The harness runs in the sandbox with the eval fixture workspace as cwd.
- Output, transcript/log evidence, generated files, and errors are captured consistently with other built-in agents.
- Documentation shows the required environment variables and an example config.
- Tests cover registry/config handling and at least a mocked or lightweight runner path.
I did not find an existing issue, PR, or code reference for PI support before opening this request.
Feature request
Add
pi-coding-agentas a supported agent harness for@vercel/agent-eval.agent-evalalready supports several coding-agent CLIs through sandboxed command execution. PI support would let users compare PI against the same fixture, result artifact, and Playground workflow used for Claude Code, Codex, Gemini, Cursor, and OpenCode.Suggested shape
pi-coding-agentorpi-cli.agentOptions, including OpenAI-compatible or Azure-style endpoint settings if the PI runtime requires them.Acceptance criteria
agent: "pi-coding-agent"oragent: "pi-cli".I did not find an existing issue, PR, or code reference for PI support before opening this request.