Generated by Architecture Council - 2026-02-01
Protocol Design Decision: WebSocket
I recommend WebSocket over TLS for the gateway protocol:
Pros:
- Native URLSessionWebSocketTask in iOS 13+
- Bidirectional real-time communication
- Built-in ping/pong for connection health
- Efficient for canvas updates and streaming
- Works through Tailscale without issues
Cons of alternatives:
- gRPC: Additional dependencies (protobuf), complex setup
- REST: Polling overhead, not suitable for real-time canvas
- MQTT: Overkill, designed for IoT pub/sub
App Architecture:
┌─────────────────────────────────────────────────────────────┐
│ OpenClaw Companion │
├─────────────────────────────────────────────────────────────┤
│ Presentation Layer (SwiftUI) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ ChatView │ │ CanvasView │ │ CapabilitySheetView │ │
│ └─────────────┘ └─────────────┘ └─────────────────────────┘│
├─────────────────────────────────────────────────────────────┤
│ Domain Layer │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ ChatManager │ │CanvasEngine │ │CapabilityController │ │
│ └─────────────┘ └─────────────┘ └─────────────────────────┘│
├─────────────────────────────────────────────────────────────┤
│ Data Layer (OpenClawKit Package) │
│ ┌─────────────────────────────────────────────────────────┐│
│ │ GatewayClient (WebSocket) │ SessionStore │ KeychainMgr ││
│ └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘
Concurrency Pattern:
- Swift Concurrency (async/await) throughout
- Actors for shared state (
GatewayActor,CapabilityActor) - AsyncStream for WebSocket messages
- @MainActor for all UI state
Data Flow:
// Message flow
Gateway WebSocket
→ AsyncStream<GatewayMessage>
→ GatewayClient.messages
→ ChatManager/CanvasEngine (Actors)
→ @Published state
→ SwiftUI ViewsAuthentication & Pairing:
Phase 1: QR Code Pairing
┌────────┐ ┌─────────┐ ┌─────────┐
│ Phone │────▶│ QR Scan │────▶│ Gateway │
└────────┘ └─────────┘ └─────────┘
│
▼
┌─────────────────┐
│ { │
│ endpoint: "..."│
│ fingerprint: "..."│
│ challenge: "..."│
│ } │
└─────────────────┘
Phase 2: Key Exchange
- Client generates Ed25519 keypair
- Sends public key + device info to gateway
- Gateway signs and returns device token
- Token stored in iOS Keychain (kSecAttrAccessibleAfterFirstUnlock)
Phase 3: Session Establishment
- WebSocket connects with device token in header
- Gateway validates token, returns session ID
- All further messages signed with session key
mTLS Implementation:
// URLSession with client certificate
let config = URLSessionConfiguration.default
config.tlsMinimumSupportedProtocolVersion = .TLSv12
// Client certificate from Keychain
let identity: SecIdentity = loadFromKeychain()
config.urlCredentialStorage?.set(
URLCredential(identity: identity, certificates: nil, persistence: .forSession),
for: protectionSpace
)Capability Token Design:
{
"capability": "camera.front.once",
"granted_at": "2026-02-01T12:00:00Z",
"expires_at": "2026-02-01T12:05:00Z",
"session_id": "abc123",
"nonce": "random_bytes",
"signature": "gateway_signature"
}Sandboxing:
- All gateway data isolated in App Group container
- No cross-app communication except via Gateway
- Background refresh disabled for sensitive capabilities
- Screen recording detection → auto-revoke camera capability
Attention Model Implementation:
┌─────────────────────────────────────────┐
│ Attention Indicator │
├─────────────────────────────────────────┤
│ PASSIVE │ ○ ○ ○ ○ │ No indicator │
│ NOTIFY │ ● ○ ○ ○ │ Badge only │
│ ACTIVE │ ● ● ○ ○ │ Subtle glow │
│ INTERRUPT│ ● ● ● ○ │ Modal appears │
│ URGENT │ ● ● ● ● │ Haptic + sound │
└─────────────────────────────────────────┘
Panic Abort Design:
- Triple-tap anywhere → Immediate session termination
- Shake gesture → Quick capability revoke sheet
- Volume buttons (long press) → Emergency disconnect
- Visual: Red pulsing border when agent has control
Capability Grant Flow (User-First):
┌────────────────────────────────────────┐
│ 🎤 Microphone Access Request │
├────────────────────────────────────────┤
│ │
│ Agent "Claude" is requesting: │
│ • Microphone access │
│ • Duration: 60 seconds │
│ • Purpose: Voice transcription │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Deny │ │ Allow 60s │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ [ ] Remember for this session │
│ │
│ ⓘ You can revoke anytime by shaking │
└────────────────────────────────────────┘
Canvas Interaction Model:
- Tap: Select/activate element
- Long press: Context menu
- Two-finger pan: Navigate canvas
- Pinch: Zoom
- Draw mode: Explicit toggle (pencil icon)
Watch UI:
┌─────────────────────┐
│ 🟢 Claude Active │
│ │
│ "Checking your │
│ calendar..." │
│ │
│ [🎤 Speak] │
│ │
│ ⚡ 2 pending │
└─────────────────────┘
Gateway Protocol (WebSocket Messages):
// Client → Gateway
interface ClientMessage {
type: "text" | "voice" | "canvas_event" | "capability_response" | "heartbeat";
session_id: string;
timestamp: number;
payload: Record<string, unknown>;
}
// Gateway → Client
interface GatewayMessage {
type: "text" | "canvas_update" | "capability_request" | "attention" | "status";
agent_id: string;
attention_level: "passive" | "notify" | "active" | "interrupt" | "urgent";
payload: Record<string, unknown>;
}
// Canvas Commands
interface CanvasCommand {
action: "clear" | "draw" | "update" | "animate";
elements: CanvasElement[];
transition?: "instant" | "fade" | "slide";
}
interface CanvasElement {
id: string;
type: "text" | "image" | "shape" | "video" | "highlight";
bounds: Rect;
content: string | Data;
style?: Style;
interactive?: boolean;
}Multimodal Input Pipeline:
Voice Input:
Microphone → AVAudioEngine → Speech.framework (local)
→ Text + confidence → Gateway
OR (streaming):
Microphone → AVAudioEngine → Opus encode
→ WebSocket binary frames → Gateway
Drawing Input:
PencilKit PKDrawing → Rasterize to PNG
→ Base64 → Gateway
Camera:
AVCaptureSession → CVPixelBuffer → JPEG compress
→ Base64 or separate upload endpoint → Gateway
Canvas Rendering Strategy:
Option A: SwiftUI Canvas (Recommended for MVP)
- Native, no dependencies
- Good for 2D graphics, text, images
- Limited 3D support
Option B: SpriteKit (If heavy animation needed)
- Better for particle effects, complex animations
- More memory overhead
Option C: Metal (Future, not MVP)
- For 3D, shaders, AR integration
Recommendation: SwiftUI Canvas for MVP, with ability to embed SpriteKit scenes for specific animations.
App Store Compliance Analysis:
| Feature | App Store Risk | Mitigation |
|---|---|---|
| AI Chat | ✅ Low | Common pattern (ChatGPT, etc.) |
| Camera Access | ✅ Low | Clear purpose string |
| Background Location | Use "While Using" only | |
| Remote Control | User must initiate all actions | |
| Canvas Scripting | No arbitrary code execution | |
| Sensor Streaming | Clear disclosure in UI | |
| BLE Scanning | 🔴 High | Only with explicit use case |
Required Privacy Strings (Info.plist):
<key>NSCameraUsageDescription</key>
<string>Share photos and video with your AI assistant</string>
<key>NSMicrophoneUsageDescription</key>
<string>Send voice messages to your AI assistant</string>
<key>NSLocationWhenInUseUsageDescription</key>
<string>Share your location with your AI assistant when asked</string>
<key>NSBluetoothAlwaysUsageDescription</key>
<string>Connect to nearby devices when requested by your assistant</string>Feature Gating Strategy:
enum DistributionChannel {
case appStore // Conservative
case testFlight // Extended
case enterprise // Full
var allowedCapabilities: Set<Capability> {
switch self {
case .appStore:
return [.camera, .microphone, .locationWhenInUse, .canvas]
case .testFlight:
return appStore.union([.backgroundLocation, .ble, .screenRecording])
case .enterprise:
return Set(Capability.allCases)
}
}
}TestFlight Strategy:
- Submit conservative build to App Store review
- Enable additional features via remote config for TestFlight users
- Monitor for policy violations before wider release
- Document all capabilities clearly in App Review notes
| Question | Decision | Rationale |
|---|---|---|
| Protocol | WebSocket/TLS | Native support, bidirectional, efficient |
| Auth | Ed25519 + JWT-like tokens | Modern, fast, compact |
| Canvas | SwiftUI Canvas | Native, sufficient for MVP |
| Watch Comms | WatchConnectivity | Only Apple-approved method |
| State | Swift Actors | Thread-safe, modern |
| Storage | Keychain + UserDefaults | Secure for tokens, fast for prefs |
- QR code pairing
- WebSocket connection
- Text chat (send/receive)
- Basic canvas (text + images)
- Capability request/grant UI
- Watch status glance
- Panic abort (shake)
- Voice input (PTT)
- Canvas drawing input
- Camera capture
- Multiple agent support
- Watch voice input
- Video playback in canvas
- Streaming voice (duplex)
- BLE device control
- Background location (TestFlight)
- Canvas animations
OpenClawCompanion/
├── App/
│ ├── OpenClawCompanionApp.swift
│ ├── Views/
│ │ ├── Chat/
│ │ ├── Canvas/
│ │ ├── Settings/
│ │ └── Pairing/
│ ├── ViewModels/
│ └── Resources/
├── Watch/
│ ├── OpenClawWatchApp.swift
│ ├── Views/
│ └── Complications/
├── Shared/
│ ├── Models/
│ └── Extensions/
├── Packages/
│ └── OpenClawKit/
│ ├── Sources/
│ │ ├── Networking/
│ │ ├── Security/
│ │ └── Protocol/
│ └── Tests/
├── project.yml # XcodeGen
└── README.md
Document version: 1.0 Next review: After MVP implementation