Skip to content

Commit 860375b

Browse files
AnnaXWangAnnaXWang
authored
Add Computer Use overview page (#425)
Introduce a landing page for the Computer Use integrations group that explains the screenshot action-observation loop, links each supported model, and points to the Computer Controls API. Wire it in as the first page of the group and redirect the bare section path to it. Document the @onkernel/cua-agent library path in the Computer Use overview with an install, a starter snippet, and a provider/model-ref table, and cross-link it from each cua-supported provider page. Replace the cross-link one-liner on each cua-supported provider page with a provider-specific CuaAgent starter snippet; install instructions stay in the overview.
1 parent 84923e8 commit 860375b

7 files changed

Lines changed: 229 additions & 2 deletions

File tree

‎docs.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
{ "source": "/auth/agent/programmatic", "destination": "/auth/programmatic" },
1212
{ "source": "/auth/agent/faq", "destination": "/auth/faq" },
1313
{ "source": "/browsers/hardware-acceleration", "destination": "/browsers/gpu-acceleration" },
14+
{ "source": "/integrations/computer-use", "destination": "/integrations/computer-use/overview" },
1415
{ "source": "/browsers/create-a-browser", "destination": "/introduction/create" },
1516
{ "source": "/introduction", "destination": "/" },
1617
{ "source": "/quickstart", "destination": "/" },
@@ -206,6 +207,7 @@
206207
{
207208
"group": "Computer Use",
208209
"pages": [
210+
"integrations/computer-use/overview",
209211
"integrations/computer-use/anthropic",
210212
"integrations/computer-use/gemini",
211213
"integrations/computer-use/openagi",

‎integrations/computer-use/anthropic.mdx‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,29 @@ Choose `TypeScript` or `Python` as the programming language.
1818

1919
Then follow the [deploy](/apps/deploy) and [invoke](/apps/invoke) guides to deploy and run your Computer Use automation on Kernel's infrastructure.
2020

21+
## Build your own agent
22+
23+
For full control over the loop, drive Claude from TypeScript with [`@onkernel/cua-agent`](/integrations/computer-use/overview#build-your-own-agent):
24+
25+
```ts
26+
import Kernel from "@onkernel/sdk";
27+
import { CuaAgent } from "@onkernel/cua-agent";
28+
29+
const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
30+
const browser = await client.browsers.create({ stealth: true });
31+
32+
const agent = new CuaAgent({
33+
browser,
34+
client,
35+
initialState: {
36+
model: "anthropic:claude-opus-4-7",
37+
systemPrompt: "You are a careful browser automation agent.",
38+
},
39+
});
40+
41+
await agent.prompt("Open news.ycombinator.com and summarize the top story.");
42+
```
43+
2144
## Benefits of using Kernel with Computer Use
2245

2346
- **No local browser management**: Run Computer Use automations without installing or maintaining browsers locally

‎integrations/computer-use/gemini.mdx‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: "Gemini"
33
---
44

5-
[Gemini 2.5 Computer Use](https://blog.google/technology/google-deepmind/gemini-computer-use-model/) is Google's groundbreaking capability that enables AI models to interact with computers the way humans do—by looking at screens, moving cursors, clicking buttons, and typing text. This powerful feature allows AI agents to control web browsers, navigate interfaces, and perform complex tasks across applications.
5+
[Gemini 2.5 Computer Use](https://blog.google/technology/google-deepmind/gemini-computer-use-model/) is Google's groundbreaking capability that enables AI models to interact with computers the way humans do by looking at screens, moving cursors, clicking buttons, and typing text. This powerful feature allows AI agents to control web browsers, navigate interfaces, and perform complex tasks across applications.
66

77
By integrating Gemini 2.5 Computer Use with Kernel, you can run these AI-powered browser automations on cloud-hosted infrastructure, eliminating the need for local browser management and enabling scalable, reliable AI agents.
88

@@ -16,6 +16,29 @@ kernel create --name my-computer-use-app --language typescript --template gemini
1616

1717
Then follow the [deploy](/apps/deploy) and [invoke](/apps/invoke) guides to deploy and run your Computer Use automation on Kernel's infrastructure.
1818

19+
## Build your own agent
20+
21+
For full control over the loop, drive Gemini from TypeScript with [`@onkernel/cua-agent`](/integrations/computer-use/overview#build-your-own-agent):
22+
23+
```ts
24+
import Kernel from "@onkernel/sdk";
25+
import { CuaAgent } from "@onkernel/cua-agent";
26+
27+
const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
28+
const browser = await client.browsers.create({ stealth: true });
29+
30+
const agent = new CuaAgent({
31+
browser,
32+
client,
33+
initialState: {
34+
model: "google:gemini-3-flash-preview",
35+
systemPrompt: "You are a careful browser automation agent.",
36+
},
37+
});
38+
39+
await agent.prompt("Open news.ycombinator.com and summarize the top story.");
40+
```
41+
1942
## Benefits of using Kernel with Computer Use
2043

2144
- **No local browser management**: Run Computer Use automations without installing or maintaining browsers locally

‎integrations/computer-use/openai.mdx‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: "OpenAI"
33
---
44

5-
[Computer Use](https://openai.com/index/computer-using-agent/) is OpenAI's feature that enables AI models to interact with computers the way humans do—by looking at screens, moving cursors, clicking buttons, and typing text. This powerful feature allows AI agents to control web browsers, navigate interfaces, and perform complex tasks across applications.
5+
[Computer Use](https://openai.com/index/computer-using-agent/) is OpenAI's feature that enables AI models to interact with computers the way humans do by looking at screens, moving cursors, clicking buttons, and typing text. This powerful feature allows AI agents to control web browsers, navigate interfaces, and perform complex tasks across applications.
66

77
By integrating Computer Use with Kernel, you can run these AI-powered browser automations on cloud-hosted infrastructure, eliminating the need for local browser management and enabling scalable, reliable AI agents.
88

@@ -18,6 +18,29 @@ Choose `TypeScript` or `Python` as the programming language.
1818

1919
Then follow the [deploy](/apps/deploy) and [invoke](/apps/invoke) guides to deploy and run your Computer Use automation on Kernel's infrastructure.
2020

21+
## Build your own agent
22+
23+
For full control over the loop, drive OpenAI's CUA from TypeScript with [`@onkernel/cua-agent`](/integrations/computer-use/overview#build-your-own-agent):
24+
25+
```ts
26+
import Kernel from "@onkernel/sdk";
27+
import { CuaAgent } from "@onkernel/cua-agent";
28+
29+
const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
30+
const browser = await client.browsers.create({ stealth: true });
31+
32+
const agent = new CuaAgent({
33+
browser,
34+
client,
35+
initialState: {
36+
model: "openai:gpt-5.5",
37+
systemPrompt: "You are a careful browser automation agent.",
38+
},
39+
});
40+
41+
await agent.prompt("Open news.ycombinator.com and summarize the top story.");
42+
```
43+
2144
## Benefits of using Kernel with Computer Use
2245

2346
- **No local browser management**: Run Computer Use automations without installing or maintaining browsers locally

‎integrations/computer-use/overview.mdx‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
title: "Overview"
3+
description: "Run computer use agents on Kernel cloud browsers"
4+
---
5+
6+
Computer use models are vision-language models (VLMs) that operate a browser the way a person does: they look at a screenshot, decide what to do next, and emit a concrete action: move the mouse, click, type, scroll, or drag. Kernel runs these agents on cloud browsers, so you don't install or maintain anything locally, and gives the model the low-level [Computer Controls API](/browsers/computer-controls) it needs to see the screen and act on it.
7+
8+
## How computer use works on Kernel
9+
10+
Every computer use integration runs the same action-observation loop:
11+
12+
1. **Capture** a screenshot of the current browser state with the [Computer Controls API](/browsers/computer-controls#take-screenshots).
13+
2. **Predict** the next action by sending that screenshot to your model.
14+
3. **Execute** the returned action (click, type, scroll, drag, or key press) through Computer Controls.
15+
4. **Repeat** until the task is complete.
16+
17+
Computer Controls emulates native keyboard and mouse input at the OS level (with human-like [Bézier curves](/browsers/computer-controls#move-the-mouse) by default) instead of driving the page over the Chrome DevTools Protocol (CDP). This keeps the loop close to real user input and reduces the automation signals that [bot detection](/browsers/bot-detection/overview) systems look for.
18+
19+
The loop works with any VLM that predicts actions from pixels. The models below are the ones we maintain ready-to-deploy templates and guides for.
20+
21+
## Supported models
22+
23+
<CardGroup cols={2}>
24+
<Card title="Anthropic" icon="robot" href="/integrations/computer-use/anthropic">
25+
Claude's computer use tool
26+
</Card>
27+
<Card title="Gemini" icon="google" href="/integrations/computer-use/gemini">
28+
Google's Gemini 2.5 Computer Use model
29+
</Card>
30+
<Card title="OpenAGI" icon="wand-magic-sparkles" href="/integrations/computer-use/openagi">
31+
OpenAGI's Lux model
32+
</Card>
33+
<Card title="OpenAI" icon="circle-nodes" href="/integrations/computer-use/openai">
34+
OpenAI's computer-using agent (CUA)
35+
</Card>
36+
<Card title="Tzafon" icon="bolt" href="/integrations/computer-use/tzafon">
37+
Tzafon's Northstar CUA Fast model
38+
</Card>
39+
<Card title="Yutori" icon="location-arrow" href="/integrations/computer-use/yutori">
40+
Yutori's Navigator n1.5 pixels-to-actions model
41+
</Card>
42+
</CardGroup>
43+
44+
Using a model that isn't listed here? Any VLM works; wire its predicted actions straight to the [Computer Controls API](/browsers/computer-controls) and run the same loop.
45+
46+
## Get started
47+
48+
Each model page includes a one-command template so you can deploy a working agent in minutes. For example, to scaffold the Anthropic integration:
49+
50+
```bash
51+
kernel create --name my-computer-use-app --template computer-use
52+
```
53+
54+
Pick a model above to get its template, then follow the [deploy](/apps/deploy) and [invoke](/apps/invoke) guides to run your agent on Kernel.
55+
56+
## Build your own agent
57+
58+
For full control over the loop, [`@onkernel/cua-agent`](https://github.com/kernel/cua/tree/main/packages/agent) is a TypeScript library that runs it against a Kernel browser for you. You point it at a model, give it a task, and it handles the screenshots, actions, and follow-up turns.
59+
60+
```bash
61+
npm install @onkernel/cua-agent @onkernel/cua-ai @onkernel/sdk
62+
```
63+
64+
```ts
65+
import Kernel from "@onkernel/sdk";
66+
import { CuaAgent } from "@onkernel/cua-agent";
67+
68+
const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
69+
const browser = await client.browsers.create({ stealth: true });
70+
71+
const agent = new CuaAgent({
72+
browser,
73+
client,
74+
initialState: {
75+
model: "anthropic:claude-opus-4-7", // swap to target another provider
76+
systemPrompt: "You are a careful browser automation agent.",
77+
},
78+
});
79+
80+
await agent.prompt("Open news.ycombinator.com and summarize the top story.");
81+
```
82+
83+
Switch providers by changing the `model` ref:
84+
85+
| Provider | Model ref |
86+
| --- | --- |
87+
| Anthropic | `anthropic:claude-opus-4-7` |
88+
| OpenAI | `openai:gpt-5.5` |
89+
| Gemini | `google:gemini-3-flash-preview` |
90+
| Tzafon | `tzafon:tzafon.northstar-cua-fast` |
91+
| Yutori | `yutori:n1.5-latest` |
92+
93+
Set the matching provider key (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `TZAFON_API_KEY`, or `YUTORI_API_KEY`) alongside `KERNEL_API_KEY`.
94+
95+
## Benefits of using Kernel for computer use
96+
97+
- **No local browser management**: Run computer use automations without installing or maintaining browsers locally
98+
- **Scalability**: Launch multiple browser sessions in parallel for concurrent AI agents
99+
- **Stealth mode**: Built-in anti-detection features for reliable web interactions
100+
- **Session state**: Maintain browser state across runs via [Profiles](/auth/profiles)
101+
- **Live view**: Debug your agents with real-time browser viewing
102+
- **Cloud infrastructure**: Run computationally intensive AI agents without local resource constraints
103+
104+
## Next steps
105+
106+
- Read the [Computer Controls API](/browsers/computer-controls) reference for the full set of mouse, keyboard, and screenshot actions
107+
- Check out [live view](/browsers/live-view) for debugging your automations
108+
- Learn about [stealth mode](/browsers/bot-detection/stealth) for avoiding detection
109+
- Learn how to properly [terminate browser sessions](/browsers/termination)
110+
- Learn how to [deploy](/apps/deploy) your computer use app to Kernel

‎integrations/computer-use/tzafon.mdx‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,29 @@ Choose `TypeScript` or `Python` as the programming language.
1818

1919
Then follow the [deploy](/apps/deploy) and [invoke](/apps/invoke) guides to deploy and run your Tzafon automation on Kernel's infrastructure.
2020

21+
## Build your own agent
22+
23+
For full control over the loop, drive Northstar CUA Fast from TypeScript with [`@onkernel/cua-agent`](/integrations/computer-use/overview#build-your-own-agent):
24+
25+
```ts
26+
import Kernel from "@onkernel/sdk";
27+
import { CuaAgent } from "@onkernel/cua-agent";
28+
29+
const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
30+
const browser = await client.browsers.create({ stealth: true });
31+
32+
const agent = new CuaAgent({
33+
browser,
34+
client,
35+
initialState: {
36+
model: "tzafon:tzafon.northstar-cua-fast",
37+
systemPrompt: "You are a careful browser automation agent.",
38+
},
39+
});
40+
41+
await agent.prompt("Open news.ycombinator.com and summarize the top story.");
42+
```
43+
2144
## Benefits of using Kernel with Tzafon Northstar CUA Fast
2245

2346
- **No local browser management**: Run Northstar CUA Fast automations without installing or maintaining browsers locally

‎integrations/computer-use/yutori.mdx‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,29 @@ Choose `TypeScript` or `Python` as the programming language.
1818

1919
Then follow the [deploy](/apps/deploy) and [invoke](/apps/invoke) guides to deploy and run your Yutori automation on Kernel's infrastructure.
2020

21+
## Build your own agent
22+
23+
For full control over the loop, drive Navigator n1.5 from TypeScript with [`@onkernel/cua-agent`](/integrations/computer-use/overview#build-your-own-agent):
24+
25+
```ts
26+
import Kernel from "@onkernel/sdk";
27+
import { CuaAgent } from "@onkernel/cua-agent";
28+
29+
const client = new Kernel({ apiKey: process.env.KERNEL_API_KEY! });
30+
const browser = await client.browsers.create({ stealth: true });
31+
32+
const agent = new CuaAgent({
33+
browser,
34+
client,
35+
initialState: {
36+
model: "yutori:n1.5-latest",
37+
systemPrompt: "You are a careful browser automation agent.",
38+
},
39+
});
40+
41+
await agent.prompt("Open news.ycombinator.com and summarize the top story.");
42+
```
43+
2144
## Benefits of using Kernel with Yutori n1.5
2245

2346
- **No local browser management**: Run n1.5 automations without installing or maintaining browsers locally

0 commit comments

Comments
 (0)