Add Anthropic (Claude) backend for AI action generation

[NakaokaRei]

Summary

Add an AnthropicBackend that uses the Claude API to convert natural language prompts into automation actions, following the same pattern as the existing OpenAIBackend.

Approach

Raw URLSession (No New Dependency)

There is no official Anthropic Swift SDK. Rather than adding a community-maintained third-party package, use raw URLSession to call the Claude Messages API (POST /v1/messages). This keeps Package.swift unchanged and avoids maintenance/compatibility risk with Swift 6.2 / macOS 26.

Structured Output via Tool Use

Claude doesn't have OpenAI-style "Structured Outputs". Instead, use tool use with forced tool choice:

• Define an execute_actions tool with the same flat-object JSON schema used by OpenAIBackend
• Set tool_choice: {"type": "tool", "name": "execute_actions"} to guarantee structured JSON output
• Extract the tool call's input and decode as ActionsWrapper → [BasicAction] → [Action]

Default Model

claude-haiku-4-5-20251001 — fast and cost-effective for this structured output task. Users can override via the model parameter.

Implementation

New file: Sources/SwiftAutoGUI/AnthropicBackend.swift

• AnthropicBackend: ActionGenerating, Sendable struct
• Properties: apiKey, model (configurable), baseURL (for testability/proxy)
• Private Codable types for HTTP request/response
• Same system prompt as OpenAIBackend
• HTTP headers: x-api-key, anthropic-version: 2023-06-01, content-type: application/json
• Error handling for HTTP status codes (401, 429, 5xx)

Modified files

• ActionGenerator.swift — Add init(anthropicKey:model:) convenience initializer
• ActionGenerating.swift — Update doc comments to list AnthropicBackend
• ActionGeneratorTests.swift — Add availability tests, convenience init test, and tool_use response JSON parsing tests
• CLAUDE.md — Add AnthropicBackend.swift to AI Action Generation section

No changes needed

• Package.swift — No new dependencies (URLSession only)

Usage

// With default model (claude-haiku-4-5-20251001)
let generator = ActionGenerator(anthropicKey: "sk-ant-...")
let actions = try await generator.generateActionSequence(from: "click at 100, 200")
await actions.execute()

// With custom model
let backend = AnthropicBackend(apiKey: "sk-ant-...", model: "claude-sonnet-4-6-20250514")
let generator = ActionGenerator(backend: backend)

[m13v]

Nice approach using tool_use with forced tool_choice for structured output from Claude. We do the same thing in our macOS automation agent and it works really well. One tip from production - when you force tool_choice to a specific tool name, Claude almost never returns malformed JSON, but it can occasionally include extra fields not in your schema. Worth adding a strict decode that ignores unknown keys rather than failing. Also for the URLSession approach, make sure to handle the anthropic-version header carefully. The 2023-06-01 version works but newer versions have slightly different tool_use response shapes. We pin ours and update deliberately. For the model choice, claude-haiku-4-5 is great for structured extraction like action sequences. We use it for similar structured output tasks and it keeps latency under 500ms for most calls.

[m13v]

Here is how we wire up the Claude provider in our Swift macOS app, including the tool_use pattern for structured responses: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatProvider.swift

[NakaokaRei]

Enhancing Screen Recognition with Vision LLM

When implementing AnthropicBackend, we can also leverage Claude's Vision capabilities to enable more advanced GUI automation.

Overview

Using the Claude Vision API, we can send screenshots to understand the current screen state and generate contextual actions accordingly. The current ActionGenerating protocol only accepts text prompts as input, but by combining screenshots, we can achieve context-aware action generation based on the actual screen state.

Proposed Extensions

1. Extend ActionGenerating protocol to accept screenshot images as input

   • e.g. func generate(from prompt: String, screenshot: CGImage?) async throws -> [Action]

2. Claude Vision API usage:

   • Send screenshots as base64-encoded image content blocks
   • Supported formats: JPEG, PNG, GIF, WebP
   • Optimal size: long edge ≤ 1568px (auto-resized otherwise, but pre-resizing improves TTFT)
   • Up to 600 images per request supported

3. Use cases:

   • "Click the red button on screen" → Identify button coordinates from the screenshot and generate a click action
   • "Fill in this form" → Recognize the form structure on screen and generate input actions
   • Combine with the existing OpenCV-based ImageRecognition for precise coordinate targeting

Notes

• Claude Vision documentation
• Claude Vision is image understanding only (no image generation)
• Spatial reasoning has limitations, so combining with OpenCV is recommended for precise coordinate detection

[m13v]

love the vision approach for screen understanding. combining screenshot analysis with the existing action generation protocol gives you both structured element interaction and visual context awareness. the ActionGeneratingWithScreenshot protocol extension is a clean way to keep it backward compatible. we found that sending screenshots at lower resolution (like 1024px width) keeps latency reasonable while still giving Claude enough detail for most UI elements.

[NakaokaRei]

we found that sending screenshots at lower resolution (like 1024px width) keeps latency reasonable while still giving Claude enough detail for most UI elements.

@m13v Thanks for your advice! Good to know 😄