Summary
Add Ollama as a fourth AI backend alongside FoundationModelsBackend, OpenAIBackend, and OpenAIVisionBackend, giving users a free, private, offline alternative for both ActionGenerator and the Agent loop. Ollama exposes an OpenAI-compatible /v1/chat/completions endpoint, so no SDK dependency is required.
Approach: extract shared logic first
Today's two OpenAI backends each contain ~30 lines of duplicated HTTP transport, envelope parsing, and action parsing. Naively adding two more files would 2x that duplication. Instead, this work will:
- Extract shared HTTP / JSON / parsing logic into
Sources/SwiftAutoGUI/Internal/ (no public API change):
ChatHTTP — POST + Bearer auth + status check + JSON envelope parse
JSONUtilities — robust JSON extraction (handles SSE / trailing garbage)
ActionDictParser — action dict -> BasicAction
ActionContentParser — inner JSON content -> [Action] / AgentResponse
ResponsesAPIDecoder — extract content from OpenAI Responses API envelope
ChatCompletionsDecoder — extract content from Chat Completions envelope
ActionSchemas — JSON schemas (action item, agent response wrapper)
SystemPrompts — actionGeneration and agentLoop(...) prompts
- Refactor
OpenAIBackend and OpenAIVisionBackend to thin wrappers over these helpers (file size drops by ~60%).
- Add
OllamaBackend and OllamaVisionBackend as similarly thin wrappers — provider-specific code is just message building + endpoint URL.
After the refactor, each concrete backend file contains only what is genuinely provider-specific.
Scope
OllamaBackend conforming to ActionGenerating (text -> action sequence).
OllamaVisionBackend conforming to VisionActionGenerating (screenshot + history -> agent response).
- Default base URL
http://localhost:11434/v1; optional apiKey parameter for proxied/cloud Ollama deployments.
--backend ollama flag added to the sagui agent CLI.
- Backend selector added to the Sample app (
AgentDemoView / AgentDemoViewModel).
init(ollamaModel:baseURL:apiKey:) convenience init on ActionGenerator.
- Updates to
README.md and DocC documentation (Documentation.docc/SwiftAutoGUI.md).
- Unit tests for the new shared helpers and Ollama-specific request building (no network).
Out of scope
- Streaming responses.
- Auto-detection of installed Ollama models.
- Pulling models from the Ollama library.
Verification
swift build — package compiles after refactor.
swift test — pre-existing tests pass; new SharedHelpersTests and OllamaBackendTests pass.
swift package generate-documentation — DocC builds without missing symbols.
- Manual smoke test with a local Ollama install:
ollama pull llama3.2-vision
swift run sagui agent "open Finder" --backend ollama --model llama3.2-vision
- Sanity-check the OpenAI path still works after the refactor (regression catch).
Notes
- No
Package.swift dependency is added — Ollama's OpenAI-compatible endpoint is consumed via URLSession, matching the existing OpenAI backends.
Summary
Add Ollama as a fourth AI backend alongside
FoundationModelsBackend,OpenAIBackend, andOpenAIVisionBackend, giving users a free, private, offline alternative for bothActionGeneratorand theAgentloop. Ollama exposes an OpenAI-compatible/v1/chat/completionsendpoint, so no SDK dependency is required.Approach: extract shared logic first
Today's two OpenAI backends each contain ~30 lines of duplicated HTTP transport, envelope parsing, and action parsing. Naively adding two more files would 2x that duplication. Instead, this work will:
Sources/SwiftAutoGUI/Internal/(no public API change):ChatHTTP— POST + Bearer auth + status check + JSON envelope parseJSONUtilities— robust JSON extraction (handles SSE / trailing garbage)ActionDictParser— action dict ->BasicActionActionContentParser— inner JSON content ->[Action]/AgentResponseResponsesAPIDecoder— extract content from OpenAI Responses API envelopeChatCompletionsDecoder— extract content from Chat Completions envelopeActionSchemas— JSON schemas (action item, agent response wrapper)SystemPrompts—actionGenerationandagentLoop(...)promptsOpenAIBackendandOpenAIVisionBackendto thin wrappers over these helpers (file size drops by ~60%).OllamaBackendandOllamaVisionBackendas similarly thin wrappers — provider-specific code is just message building + endpoint URL.After the refactor, each concrete backend file contains only what is genuinely provider-specific.
Scope
OllamaBackendconforming toActionGenerating(text -> action sequence).OllamaVisionBackendconforming toVisionActionGenerating(screenshot + history -> agent response).http://localhost:11434/v1; optionalapiKeyparameter for proxied/cloud Ollama deployments.--backend ollamaflag added to thesagui agentCLI.AgentDemoView/AgentDemoViewModel).init(ollamaModel:baseURL:apiKey:)convenience init onActionGenerator.README.mdand DocC documentation (Documentation.docc/SwiftAutoGUI.md).Out of scope
Verification
swift build— package compiles after refactor.swift test— pre-existing tests pass; newSharedHelpersTestsandOllamaBackendTestspass.swift package generate-documentation— DocC builds without missing symbols.ollama pull llama3.2-vision swift run sagui agent "open Finder" --backend ollama --model llama3.2-visionNotes
Package.swiftdependency is added — Ollama's OpenAI-compatible endpoint is consumed viaURLSession, matching the existing OpenAI backends.