Skip to content

feat: add Ollama backend (text + vision) with shared transport refactor #90

Description

@NakaokaRei

Summary

Add Ollama as a fourth AI backend alongside FoundationModelsBackend, OpenAIBackend, and OpenAIVisionBackend, giving users a free, private, offline alternative for both ActionGenerator and the Agent loop. Ollama exposes an OpenAI-compatible /v1/chat/completions endpoint, so no SDK dependency is required.

Approach: extract shared logic first

Today's two OpenAI backends each contain ~30 lines of duplicated HTTP transport, envelope parsing, and action parsing. Naively adding two more files would 2x that duplication. Instead, this work will:

  1. Extract shared HTTP / JSON / parsing logic into Sources/SwiftAutoGUI/Internal/ (no public API change):
    • ChatHTTP — POST + Bearer auth + status check + JSON envelope parse
    • JSONUtilities — robust JSON extraction (handles SSE / trailing garbage)
    • ActionDictParser — action dict -> BasicAction
    • ActionContentParser — inner JSON content -> [Action] / AgentResponse
    • ResponsesAPIDecoder — extract content from OpenAI Responses API envelope
    • ChatCompletionsDecoder — extract content from Chat Completions envelope
    • ActionSchemas — JSON schemas (action item, agent response wrapper)
    • SystemPromptsactionGeneration and agentLoop(...) prompts
  2. Refactor OpenAIBackend and OpenAIVisionBackend to thin wrappers over these helpers (file size drops by ~60%).
  3. Add OllamaBackend and OllamaVisionBackend as similarly thin wrappers — provider-specific code is just message building + endpoint URL.

After the refactor, each concrete backend file contains only what is genuinely provider-specific.

Scope

  • OllamaBackend conforming to ActionGenerating (text -> action sequence).
  • OllamaVisionBackend conforming to VisionActionGenerating (screenshot + history -> agent response).
  • Default base URL http://localhost:11434/v1; optional apiKey parameter for proxied/cloud Ollama deployments.
  • --backend ollama flag added to the sagui agent CLI.
  • Backend selector added to the Sample app (AgentDemoView / AgentDemoViewModel).
  • init(ollamaModel:baseURL:apiKey:) convenience init on ActionGenerator.
  • Updates to README.md and DocC documentation (Documentation.docc/SwiftAutoGUI.md).
  • Unit tests for the new shared helpers and Ollama-specific request building (no network).

Out of scope

  • Streaming responses.
  • Auto-detection of installed Ollama models.
  • Pulling models from the Ollama library.

Verification

  • swift build — package compiles after refactor.
  • swift test — pre-existing tests pass; new SharedHelpersTests and OllamaBackendTests pass.
  • swift package generate-documentation — DocC builds without missing symbols.
  • Manual smoke test with a local Ollama install:
    ollama pull llama3.2-vision
    swift run sagui agent "open Finder" --backend ollama --model llama3.2-vision
  • Sanity-check the OpenAI path still works after the refactor (regression catch).

Notes

  • No Package.swift dependency is added — Ollama's OpenAI-compatible endpoint is consumed via URLSession, matching the existing OpenAI backends.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions