Skip to content

Support pi-coding-agent as an agent harness #156

Description

@christso

Feature request

Add pi-coding-agent as a supported agent harness for @vercel/agent-eval.

agent-eval already supports several coding-agent CLIs through sandboxed command execution. PI support would let users compare PI against the same fixture, result artifact, and Playground workflow used for Claude Code, Codex, Gemini, Cursor, and OpenCode.

Suggested shape

  • Add a built-in agent such as pi-coding-agent or pi-cli.
  • Prefer the same sandboxed command boundary used by existing agents: run PI as a CLI command or as a small isolated runner inside the sandbox, then capture stdout/stderr, transcript evidence, generated files, and exit status.
  • Avoid importing a long-running agent SDK directly into the evaluator parent process if a CLI/child-process boundary can provide the same behavior. Child failures should become agent-run failures, not evaluator-process crashes.
  • Support model/provider configuration through agentOptions, including OpenAI-compatible or Azure-style endpoint settings if the PI runtime requires them.
  • Preserve the existing result bundle and Playground compatibility so PI runs can be inspected and compared with other agents.

Acceptance criteria

  • A config can select the PI harness, for example agent: "pi-coding-agent" or agent: "pi-cli".
  • The harness runs in the sandbox with the eval fixture workspace as cwd.
  • Output, transcript/log evidence, generated files, and errors are captured consistently with other built-in agents.
  • Documentation shows the required environment variables and an example config.
  • Tests cover registry/config handling and at least a mocked or lightweight runner path.

I did not find an existing issue, PR, or code reference for PI support before opening this request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions