feat(templates): Implement Gemini Computer Use templates for TypeScript and Python (#94)

# feat(templates): Implement Gemini Computer Use templates for
TypeScript and Python

## Summary

This PR introduces full-fledged Gemini Computer Use templates for both
TypeScript and Python. The templates implement Google's [Gemini 2.5
Computer Use](https://ai.google.dev/gemini-api/docs/computer-use) model
using Kernel's Computer Controls API for browser automation.

## Changes

### New Templates
- **TypeScript**: Complete rewrite of `gemini-computer-use` template
with native Kernel integration
- **Python**: New `gemini-computer-use` template following the same
architecture

### Architecture

Both templates follow a modular design based on Google's
[computer-use-preview
reference](https://github.com/google-gemini/computer-use-preview):

| File | Description |
|------|-------------|
| `index.ts` / `main.py` | Entry point with Kernel app registration and
action handler |
| `loop.ts` / `loop.py` | Gemini sampling loop - orchestrates model
calls and action execution |
| `session.ts` / `session.py` | Browser session management with optional
replay recording |
| `tools/computer.ts` / `tools/computer.py` | Maps Gemini actions to
Kernel's Computer Controls API |
| `tools/types/` | Type definitions for Gemini function calls |

### Supported Actions

| Action | Description |
|--------|-------------|
| `click_at` | Click at coordinates (x, y) |
| `hover_at` | Move mouse to coordinates |
| `type_text_at` | Click and type text at coordinates |
| `scroll_document` | Scroll page (up/down/left/right) |
| `scroll_at` | Scroll at specific coordinates |
| `navigate` | Navigate to a URL |
| `go_back` / `go_forward` | Browser history navigation |
| `key_combination` | Press key combinations (e.g., "ctrl+c") |
| `drag_and_drop` | Drag from one point to another |
| `wait_5_seconds` | Wait action |

### Key Features

- **Native Kernel Integration**: Uses Kernel's Computer Controls API
directly instead of third-party browser automation libraries
- **Replay Recording**: Optional video replay recording for debugging
(paid plans only)
- **Context Management**: Intelligently manages screenshot history to
stay within context limits
- **Coordinate Normalization**: Handles Gemini's normalized coordinates
(0-1000) to actual screen dimensions
- **Local Development**: Both templates support local execution via `npx
tsx index.ts` / direct Python execution

### CLI Updates

- Added `TemplateGeminiComputerUse` constant and template metadata in
`pkg/create/templates.go`
- Templates prioritized in selection list alongside Anthropic and OpenAI
computer use templates

## Usage

```bash
# Create a new project
kernel create my-app --language typescript --template gemini-computer-use

# Deploy
kernel deploy index.ts --env-file .env

# Invoke
kernel invoke ts-gemini-cua cua-task --payload '{"query": "Navigate to https://example.com and describe the page"}'

# With replay recording
kernel invoke ts-gemini-cua cua-task --payload '{"query": "...", "record_replay": true}'
```

## Requirements

- `GOOGLE_API_KEY` - [Google AI
Studio](https://aistudio.google.com/apikey)

## Testing

Templates have been manually tested with various browser automation
tasks.

## Related

- Closes KERNEL-870

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Adds native Gemini Computer Use support across both languages and
aligns docs/tests/CLI metadata.
> 
> - **New Python template** `python/gemini-computer-use` with `main.py`,
`loop.py`, `session.py`, and `tools/` implementing Kernel Computer
Controls and Gemini sampling loop
> - **TypeScript template rewrite** `typescript/gemini-computer-use`:
replaces Stagehand with native Kernel APIs; adds `loop.ts`,
`session.ts`, `tools/`; updates `index.ts`, README, and dependencies
> - **CLI templates registry**: marks `GeminiComputerUse` as available
for Python and TypeScript; sets concrete invoke commands for both
languages
> - **QA docs**: adds `py-gemini-cua` to matrix, create/deploy steps,
invoke commands; updates totals (18 apps/21 tests)
> - **Tests**: removes "gemini-computer-use not available for python"
cases and related unavailability checks
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5e859dc. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Author: dprevoznik

.cursor/commands/qa.md

@@ -63,6 +63,7 @@ Here are all valid language + template combinations:
 > **Note:** The `yutori-computer-use` template supports two modes: `computer_use` (default, full VM screenshots) and `playwright` (viewport-only screenshots via CDP). Both modes should be tested.
 
 | python     | sample-app             | py-sample-app     | python-basic          | No             | -                              |
+| python     | gemini-computer-use    | py-gemini-cua     | python-gemini-cua     | Yes            | GOOGLE_API_KEY                 |
 | python     | captcha-solver         | py-captcha-solver | python-captcha-solver | No             | -                              |
 | python     | browser-use            | py-browser-use    | python-bu             | Yes            | OPENAI_API_KEY                 |
 | python     | anthropic-computer-use | py-anthropic-cua  | python-anthropic-cua  | Yes            | ANTHROPIC_API_KEY              |
@@ -99,6 +100,7 @@ Run each of these (they are non-interactive when all flags are provided):
 ../bin/kernel create -n py-openai-cua -l python -t openai-computer-use
 ../bin/kernel create -n py-openagi-cua -l python -t openagi-computer-use
 ../bin/kernel create -n py-claude-agent-sdk -l python -t claude-agent-sdk
+../bin/kernel create -n py-gemini-cua -l python -t gemini-computer-use
 ../bin/kernel create -n py-yutori-cua -l python -t yutori-computer-use
 ```
 
@@ -241,6 +243,15 @@ echo "ANTHROPIC_API_KEY=<value from human>" > .env
 cd ..
 ```
 
+**py-gemini-cua** (needs GOOGLE_API_KEY):
+
+```bash
+cd py-gemini-cua
+echo "GOOGLE_API_KEY=<value from human>" > .env
+../bin/kernel deploy main.py --env-file .env
+cd ..
+```
+
 **py-yutori-cua** (needs YUTORI_API_KEY):
 
 ```bash
@@ -262,7 +273,7 @@ kernel invoke ts-stagehand teamsize-task --payload '{"company": "Kernel"}'
 kernel invoke ts-anthropic-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and move the 5 items in the To Do and In Progress items to the Done section of the Kanban board. You are done successfully when the items are moved.", "record_replay": true}'
 kernel invoke ts-magnitude mag-url-extract --payload '{"url": "https://en.wikipedia.org/wiki/Special:Random"}'
 kernel invoke ts-openai-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
-kernel invoke ts-gemini-cua gemini-cua-task --payload '{"startingUrl": "https://www.magnitasks.com/", "instruction": "Click the Tasks option in the left-side bar, and move the 5 items in the To Do and In Progress items to the Done section of the Kanban board? You are done successfully when the items are moved."}'
+kernel invoke ts-gemini-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and move the 5 items in the To Do and In Progress items to the Done section of the Kanban board. You are done successfully when the items are moved.", "record_replay": true}'
 kernel invoke ts-claude-agent-sdk agent-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 3 stories"}'
 kernel invoke ts-yutori-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "computer_use"}'
 kernel invoke ts-yutori-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "playwright"}'
@@ -275,13 +286,14 @@ kernel invoke python-anthropic-cua cua-task --payload '{"query": "Go to http://m
 kernel invoke python-openai-cua cua-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 5 articles"}'
 kernel invoke python-openagi-cua openagi-default-task -p '{"instruction": "Navigate to https://agiopen.org and click the What is Computer Use? button"}'
 kernel invoke py-claude-agent-sdk agent-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 3 stories"}'
+kernel invoke python-gemini-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and move the 5 items in the To Do and In Progress items to the Done section of the Kanban board. You are done successfully when the items are moved.", "record_replay": true}'
 kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "computer_use"}'
 kernel invoke python-yutori-cua cua-task --payload '{"query": "Go to http://magnitasks.com, Click the Tasks option in the left-side bar, and drag the 5 items in the To Do and In Progress columns to the Done section of the Kanban board. You are done successfully when the items are dragged to Done. Do not click into the items.", "record_replay": true, "mode": "playwright"}'
 ```
 
 ## Step 7: Automated Runtime Testing (Optional)
 
-**STOP and ask the human:** "Would you like me to automatically invoke all 19 test cases and report back on their runtime status?"
+**STOP and ask the human:** "Would you like me to automatically invoke all 21 test cases and report back on their runtime status?"
 
 If the human agrees, invoke each template use the Kernel CLI and collect results. Present findings in this format:
 
@@ -310,6 +322,7 @@ If the human agrees, invoke each template use the Kernel CLI and collect results
 | py-openai-cua     | python-openai-cua     |         |       |
 | py-openagi-cua    | python-openagi-cua    |         |       |
 | py-claude-agent-sdk | py-claude-agent-sdk |         |       |
+| py-gemini-cua     | python-gemini-cua     |         |       |
 | py-yutori-cua     | python-yutori-cua     |         | mode: computer_use |
 | py-yutori-cua     | python-yutori-cua     |         | mode: playwright |
 
@@ -324,9 +337,9 @@ Notes should include brief error messages for failures or confirmation of succes
 - [ ] Built CLI with `make build`
 - [ ] Created QA directory
 - [ ] Got KERNEL_API_KEY from human
-- [ ] Created all 17 template variations
+- [ ] Created all 18 template variations
 - [ ] Got required API keys from human (OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, OAGI_API_KEY, YUTORI_API_KEY)
-- [ ] Deployed all 17 apps
+- [ ] Deployed all 18 apps
 - [ ] Provided invoke commands to human for manual testing
 - [ ] (Optional) Ran automated runtime testing and reviewed results
 

cmd/create_test.go

@@ -439,12 +439,6 @@ func TestCreateCommand_InvalidLanguageTemplateCombinations(t *testing.T) {
 			template:    create.TemplateMagnitude,
 			errContains: "template not found: python/magnitude",
 		},
-		{
-			name:        "gemini-computer-use not available for python",
-			language:    create.LanguagePython,
-			template:    create.TemplateGeminiComputerUse,
-			errContains: "template not found: python/gemini-computer-use",
-		},
 		{
 			name:        "invalid language",
 			language:    "ruby",
@@ -558,7 +552,6 @@ func TestCreateCommand_TemplateNotAvailableForLanguage(t *testing.T) {
 		create.TemplateBrowserUse: {create.LanguageTypeScript},
 		create.TemplateStagehand:  {create.LanguagePython},
 		create.TemplateMagnitude:  {create.LanguagePython},
-		create.TemplateGeminiComputerUse: {create.LanguagePython},
 	}
 
 	for template, unavailableLanguages := range unavailableCombinations {

pkg/create/templates.go

@@ -63,7 +63,7 @@ var Templates = map[string]TemplateInfo{
 	TemplateGeminiComputerUse: {
 		Name:        "Gemini Computer Use",
 		Description: "Implements a Gemini computer use agent",
-		Languages:   []string{LanguageTypeScript},
+		Languages:   []string{LanguageTypeScript, LanguagePython},
 	},
 	TemplateBrowserUse: {
 		Name:        "Browser Use",
@@ -201,7 +201,7 @@ var Commands = map[string]map[string]DeployConfig{
 		TemplateGeminiComputerUse: {
 			EntryPoint:    "index.ts",
 			NeedsEnvFile:  true,
-			InvokeCommand: "kernel invoke ts-gemini-cua gemini-cua-task",
+			InvokeCommand: `kernel invoke ts-gemini-cua cua-task --payload '{"query": "Navigate to http://magnitasks.com and click on Tasks in the sidebar"}'`,
 		},
 		TemplateClaudeAgentSDK: {
 			EntryPoint:    "index.ts",
@@ -250,6 +250,11 @@ var Commands = map[string]map[string]DeployConfig{
 			NeedsEnvFile:  true,
 			InvokeCommand: `kernel invoke py-claude-agent-sdk agent-task --payload '{"task": "Go to https://news.ycombinator.com and get the top 3 stories"}'`,
 		},
+		TemplateGeminiComputerUse: {
+			EntryPoint:    "main.py",
+			NeedsEnvFile:  true,
+			InvokeCommand: `kernel invoke python-gemini-cua cua-task --payload '{"query": "Navigate to http://magnitasks.com and click on Tasks in the sidebar"}'`,
+		},
 		TemplateYutoriComputerUse: {
 			EntryPoint:    "main.py",
 			NeedsEnvFile:  true,

pkg/templates/python/gemini-computer-use/README.md

@@ -0,0 +1,59 @@
+# Kernel Python Sample App - Gemini Computer Use
+
+This is a Kernel application that implements a prompt loop using Google's Gemini Computer Use model with Kernel's Computer Controls API.
+
+## Setup
+
+1. Get your API keys:
+   - **Kernel**: [dashboard.onkernel.com](https://dashboard.onkernel.com)
+   - **Google AI**: [aistudio.google.com/api-keys](https://aistudio.google.com/api-keys)
+
+2. Deploy the app:
+```bash
+kernel login
+cp .env.example .env  # Add your GOOGLE_API_KEY
+kernel deploy main.py --env-file .env
+```
+
+## Usage
+
+```bash
+kernel invoke python-gemini-cua cua-task --payload '{"query": "Navigate to https://example.com and describe the page"}'
+```
+
+## Recording Replays
+
+> **Note:** Replay recording is only available to Kernel users on paid plans.
+
+Add `"record_replay": true` to your payload to capture a video of the browser session:
+
+```bash
+kernel invoke python-gemini-cua cua-task --payload '{"query": "Navigate to https://example.com", "record_replay": true}'
+```
+
+When enabled, the response will include a `replay_url` field with a link to view the recorded session.
+
+## Gemini Computer Use Actions
+
+The Gemini model can execute the following browser actions:
+
+| Action | Description |
+|--------|-------------|
+| `open_web_browser` | Returns a screenshot (browser is already running) |
+| `click_at` | Click at coordinates (x, y) |
+| `hover_at` | Move mouse to coordinates (x, y) |
+| `type_text_at` | Click and type text at coordinates |
+| `scroll_document` | Scroll the page (up/down/left/right) |
+| `scroll_at` | Scroll at specific coordinates |
+| `search` | Focus the browser URL bar |
+| `navigate` | Navigate to a URL |
+| `go_back` | Go back in browser history |
+| `go_forward` | Go forward in browser history |
+| `key_combination` | Press key combination (e.g., "ctrl+c") |
+| `drag_and_drop` | Drag from one point to another |
+| `wait_5_seconds` | Wait for 5 seconds |
+
+## Resources
+
+- [Google Gemini Computer Use Documentation](https://ai.google.dev/gemini-api/docs/computer-use)
+- [Kernel Computer Controls](https://www.kernel.sh/docs/browsers/computer-controls)

pkg/templates/python/gemini-computer-use/_gitignore

@@ -0,0 +1,6 @@
+.venv/
+__pycache__/
+*.pyc
+.env
+.env.local
+uv.lock