Add class-based CUA agent surfaces and runtime spec

.github/workflows/ci.yml

@@ -28,7 +28,8 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 15
     # Only run on the main repo (not forks) so secrets are available.
-    if: github.event.pull_request.head.repo.full_name == github.repository || github.event_name == 'push'
+    # Skip draft PRs; still run on every commit to non-draft PRs and on main pushes.
+    if: github.event_name == 'push' || (github.event.pull_request.head.repo.full_name == github.repository && !github.event.pull_request.draft)
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
@@ -45,3 +46,29 @@ jobs:
           TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
           YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
         run: npm test --workspace @onkernel/cua-ai -- test/batch-tool.integration.test.ts
+
+  agent-e2e:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    # Only run on the main repo (not forks) so secrets are available.
+    # Skip draft PRs; still run on every commit to non-draft PRs and on main pushes.
+    if: github.event_name == 'push' || (github.event.pull_request.head.repo.full_name == github.repository && !github.event.pull_request.draft)
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+      - run: npm ci
+      - run: npm run build --workspace @onkernel/cua-ai
+      - run: npm run build --workspace @onkernel/cua-agent
+      - name: Agent live smoke tests (all providers)
+        env:
+          CUA_E2E_LIVE: "1"
+          KERNEL_API_KEY: ${{ secrets.KERNEL_API_KEY }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+          TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
+          YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
+        run: npm test --workspace @onkernel/cua-agent -- test/e2e.live.test.ts

docs/architecture.md

@@ -5,6 +5,9 @@ someone who wants to read the code, contribute, or fork.
 
 ## Design goals and invariants
 
+- `@onkernel/cua-agent` is provider-neutral runtime glue around
+  `pi-agent-core`. It should never branch on provider identity directly.
+  Provider-specific behavior must come from `@onkernel/cua-ai`.
 - Provider packages are generic glue between a model provider and Kernel
   browsers. Their package roots expose provider-neutral helpers, tool
   specs, and execution functions that can be embedded in non-`pi` loops.
@@ -28,6 +31,25 @@ someone who wants to read the code, contribute, or fork.
   and TUI regression tests. It is part of the monorepo build graph, but
   not part of the runtime browser/model/provider path.
 
+## `cua-ai` vs `cua-agent` ownership boundary
+
+`@onkernel/cua-ai` and `@onkernel/cua-agent` intentionally split concerns:
+
+- `@onkernel/cua-ai` owns provider-specific policy:
+  - provider model refs and provider resolution
+  - provider default system prompts
+  - provider payload transforms and protocol quirks (for example, Yutori tool serialization policy)
+  - canonical CUA tool-definition exports
+- `@onkernel/cua-agent` owns browser execution orchestration:
+  - `CuaAgent` / `CuaHarness` class wiring around `pi-agent-core`
+  - executing canonical CUA tool calls against Kernel browsers
+  - typed executor coverage and translator integration
+
+In practice this means any new provider quirk should be implemented in
+`@onkernel/cua-ai` and surfaced through provider-neutral runtime specs.
+`@onkernel/cua-agent` should consume that spec without explicit
+provider-specific conditionals.
+
 ## Layers
 
 `cua` is a thin TypeScript monorepo on top of the

packages/agent/CHANGELOG.md

@@ -2,6 +2,6 @@
 
 ## 0.1.0
 
-- Initial `@onkernel/cua-agent` package.
-- Added `createCuaAgent()` returning a pi-agent-core `Agent`.
-- Added Kernel browser computer `AgentTool` constructors with internal translator plumbing.
+- Class-first CUA runtime: `CuaAgent` and `CuaHarness` on top of pi-agent-core.
+- Provider-neutral browser tool executors for canonical CUA tool names, backed by Kernel browser actions.
+- Includes examples plus unit and live e2e coverage for common provider/model combinations.

packages/agent/README.md

@@ -1,28 +1,27 @@
 # `@onkernel/cua-agent`
 
-Kernel browser computer-use tools and a small `Agent` factory built on
+Kernel browser computer-use classes built on
 [`@earendil-works/pi-agent-core`](https://github.com/earendil-works/pi/tree/main/packages/agent).
 
-The package keeps pi-agent-core's `Agent`, `AgentOptions`, `AgentTool`, event
-stream, and state model intact. Kernel only supplies the browser execution
-plumbing.
+This package keeps pi-agent-core semantics intact and adds browser execution
+plumbing for canonical CUA tools.
 
 ## Installation
 
 ```bash
 npm install @onkernel/cua-agent @onkernel/cua-ai @onkernel/sdk
 ```
 
-## Quick Start
+## Quick Start (`CuaAgent`)
 
 ```ts
 import Kernel from "@onkernel/sdk";
-import { createCuaAgent } from "@onkernel/cua-agent";
+import { CuaAgent } from "@onkernel/cua-agent";
 
 const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
 const browser = await client.browsers.create({ stealth: true });
 
-const agent = createCuaAgent({
+const agent = new CuaAgent({
   browser,
   client,
   initialState: {
@@ -34,52 +33,71 @@ const agent = createCuaAgent({
 await agent.prompt("Open news.ycombinator.com and summarize the top story.");
 ```
 
-## Core Concepts
-
-### It Returns a pi Agent
-
-`createCuaAgent()` returns the underlying pi-agent-core `Agent` directly.
-Subscribe to events, mutate `agent.state`, call `prompt()`, `continue()`,
-`steer()`, and `followUp()` the same way you would with pi-agent-core.
-
-### Configuration Lives in `initialState`
-
-The API mirrors the pi-agent-core quick start:
+## Quick Start (`CuaHarness`)
 
 ```ts
-const agent = createCuaAgent({
+import { CuaHarness } from "@onkernel/cua-agent";
+
+const harness = new CuaHarness({
   browser,
   client,
-  initialState: {
-    model: "yutori:n1.5-latest",
-    tools: myTools,
-    systemPrompt: "Use the browser to complete the task.",
-  },
+  model: "openai:gpt-5.5",
 });
+
+await harness.prompt("Open example.com and tell me the current URL.");
+const transcript = harness.getTranscript();
+console.log("messages in transcript:", transcript.length);
 ```
 
-If `initialState.tools` is omitted, Kernel installs the provider-specific CUA
-computer tools. If `initialState.tools` is provided, it is used exactly.
+Use `CuaAgent` when you want direct pi `Agent` control. Use `CuaHarness` when you
+want the harness-style constructor plus transcript-oriented helpers like
+`getTranscript()`.
+
+## Core Concepts
+
+### Class-First API
+
+- `CuaAgent extends Agent`
+- `CuaHarness` wraps a pi `Agent` with a harness-style constructor and
+  delegated runtime methods.
+
+Both classes mirror pi constructor shapes and behavior, with minimal additions:
+- `browser` (Kernel browser response)
+- `client` (Kernel SDK client)
+- CUA model refs (`"provider:model"`) accepted where pi expects a concrete model
+
+If `getApiKey` is omitted, both classes default to CUA env var conventions:
+- OpenAI: `OPENAI_API_KEY`
+- Anthropic: `ANTHROPIC_OAUTH_TOKEN` or `ANTHROPIC_API_KEY`
+- Gemini: `GOOGLE_API_KEY` or `GEMINI_API_KEY`
+- Tzafon: `TZAFON_API_KEY`
+- Yutori: `YUTORI_API_KEY`
+
+### Tool Defaults
+
+If tools are omitted, the classes install canonical CUA computer tool executors
+using runtime specs from `@onkernel/cua-ai`. If tools are provided, they are
+used exactly.
 
 ### Tool Composition
 
-Use `createCuaComputerTools()` when you want to extend the default set:
+Use `createCuaComputerTools()` to compose your own tool list from canonical
+tool definitions:
 
 ```ts
+import { resolveCuaRuntimeSpec } from "@onkernel/cua-ai";
+import { createCuaComputerTools } from "@onkernel/cua-agent";
+
+const runtime = resolveCuaRuntimeSpec("openai:gpt-5.5");
 const tools = [
-  ...createCuaComputerTools({ provider: "openai", browser, client }),
+  ...createCuaComputerTools({
+    browser,
+    client,
+    toolDefinitions: runtime.toolDefinitions,
+  }),
   myCustomTool,
 ];
 ```
 
-There are no magic tool preset strings and no bundled coding/file tools. Compose
-those yourself with pi packages or your own tools.
-
-### Browser Plumbing
-
-Public helpers accept Kernel SDK browser responses plus a Kernel client. The
-internal translator handles screenshots, coordinate conversion, URL reads, and
-Kernel computer API calls behind the scenes.
-
 For full event semantics, steering, follow-up queues, and tool execution
 details, see the pi-agent-core README.

packages/agent/examples/agent-openai-smoke.ts

@@ -0,0 +1,41 @@
+import Kernel from "@onkernel/sdk";
+import { requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai";
+import { CuaAgent } from "../src/index";
+import { SCENARIOS } from "./shared/scenarios";
+
+const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.5";
+
+async function main(): Promise<void> {
+	const kernelApiKey = process.env.KERNEL_API_KEY;
+	if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required");
+	requireCuaEnvApiKeyForModel(modelRef);
+	const client = new Kernel({ apiKey: kernelApiKey });
+	const browser = await client.browsers.create({ stealth: true });
+
+	try {
+		const agent = new CuaAgent({
+			browser,
+			client,
+			initialState: { model: modelRef },
+		});
+
+		agent.subscribe((event) => {
+			if (event.type === "tool_execution_start") {
+				console.log(`[tool:start] ${event.toolName}`);
+			}
+			if (event.type === "tool_execution_end") {
+				console.log(`[tool:end] ${event.toolName} error=${event.isError}`);
+			}
+		});
+
+		const scenario = SCENARIOS[0]!;
+		console.log(`running scenario: ${scenario.name}`);
+		await agent.prompt(scenario.prompt);
+		const assistant = [...agent.state.messages].reverse().find((message) => message.role === "assistant");
+		console.log("assistant stopReason:", assistant?.role === "assistant" ? assistant.stopReason : "unknown");
+	} finally {
+		await client.browsers.deleteByID(browser.session_id);
+	}
+}
+
+void main();

packages/agent/examples/agent-provider-matrix.ts

@@ -0,0 +1,30 @@
+import Kernel from "@onkernel/sdk";
+import { requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai";
+import { CuaAgent } from "../src/index";
+import { SCENARIOS } from "./shared/scenarios";
+
+const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.5";
+const scenarioName = process.env.SCENARIO ?? SCENARIOS[0]!.name;
+
+async function main(): Promise<void> {
+	const kernelApiKey = process.env.KERNEL_API_KEY;
+	if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required");
+	requireCuaEnvApiKeyForModel(modelRef);
+	const client = new Kernel({ apiKey: kernelApiKey });
+	const browser = await client.browsers.create({ stealth: true });
+	const scenario = SCENARIOS.find((entry) => entry.name === scenarioName) ?? SCENARIOS[0]!;
+
+	try {
+		const agent = new CuaAgent({
+			browser,
+			client,
+			initialState: { model: modelRef },
+		});
+		console.log(`model=${modelRef} scenario=${scenario.name}`);
+		await agent.prompt(scenario.prompt);
+	} finally {
+		await client.browsers.deleteByID(browser.session_id);
+	}
+}
+
+void main();

packages/agent/examples/harness-openai-smoke.ts

@@ -0,0 +1,43 @@
+import Kernel from "@onkernel/sdk";
+import { requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai";
+import { CuaHarness } from "../src/index";
+import { SCENARIOS } from "./shared/scenarios";
+
+const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.5";
+
+async function main(): Promise<void> {
+	const kernelApiKey = process.env.KERNEL_API_KEY;
+	if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required");
+	requireCuaEnvApiKeyForModel(modelRef);
+	const client = new Kernel({ apiKey: kernelApiKey });
+	const browser = await client.browsers.create({ stealth: true });
+
+	try {
+		const harness = new CuaHarness({
+			browser,
+			client,
+			model: modelRef,
+		});
+
+		harness.subscribe((event) => {
+			if (event.type === "tool_execution_start") {
+				console.log(`[tool:start] ${event.toolName}`);
+			}
+			if (event.type === "tool_execution_end") {
+				console.log(`[tool:end] ${event.toolName} error=${event.isError}`);
+			}
+		});
+
+		const scenario = SCENARIOS[0]!;
+		console.log(`running scenario: ${scenario.name}`);
+		await harness.prompt(scenario.prompt);
+		const transcript = harness.getTranscript();
+		const lastAssistant = [...transcript].reverse().find((message) => message.role === "assistant");
+		console.log("transcript messages:", transcript.length);
+		console.log("assistant stopReason:", lastAssistant?.role === "assistant" ? lastAssistant.stopReason : "unknown");
+	} finally {
+		await client.browsers.deleteByID(browser.session_id);
+	}
+}
+
+void main();

packages/agent/examples/harness-provider-matrix.ts

@@ -0,0 +1,31 @@
+import Kernel from "@onkernel/sdk";
+import { requireCuaEnvApiKeyForModel, type CuaModelRef } from "@onkernel/cua-ai";
+import { CuaHarness } from "../src/index";
+import { SCENARIOS } from "./shared/scenarios";
+
+const modelRef = (process.env.MODEL_REF as CuaModelRef | undefined) ?? "openai:gpt-5.5";
+const scenarioName = process.env.SCENARIO ?? SCENARIOS[0]!.name;
+
+async function main(): Promise<void> {
+	const kernelApiKey = process.env.KERNEL_API_KEY;
+	if (!kernelApiKey) throw new Error("KERNEL_API_KEY is required");
+	requireCuaEnvApiKeyForModel(modelRef);
+	const client = new Kernel({ apiKey: kernelApiKey });
+	const browser = await client.browsers.create({ stealth: true });
+	const scenario = SCENARIOS.find((entry) => entry.name === scenarioName) ?? SCENARIOS[0]!;
+
+	try {
+		const harness = new CuaHarness({
+			browser,
+			client,
+			model: modelRef,
+		});
+		console.log(`model=${modelRef} scenario=${scenario.name}`);
+		await harness.prompt(scenario.prompt);
+		console.log("transcript messages:", harness.getTranscript().length);
+	} finally {
+		await client.browsers.deleteByID(browser.session_id);
+	}
+}
+
+void main();