Skip to content

Add cross-platform audio backend foundation #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pranc1ngpegasus merged 2 commits into from
Aug 1, 2026
Merged

Add cross-platform audio backend foundation #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
853f028
Add cross-platform audio backend foundation
pranc1ngpegasus Jul 31, 2026
da63523
Fix Linux callback receiver lint
pranc1ngpegasus Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,34 @@ Microphone input ───────┘ │
└─► wisp-storage (SQLite + Ogg/Opus)
```

### Cross-platform audio boundary

The Rust boundary separates OS capture from transcription:

- `wisp-core` owns stable `TrackId`/`SourceKind`, device-native
`AudioFormat`/`AudioFrame`, `CaptureEvent`, and partial/final
`TranscriptEvent` contracts. The existing `SourceLabel::Mic` and
`SourceLabel::System` map to fixed track IDs, so storage and UI behavior stay
compatible while future application/process tracks remain possible.
- `wisp-audiokit` owns capability probes, `CaptureBackend` and
`TranscriberBackend`, privacy-aware backend selection, and
`SessionOrchestrator`. An offline-required policy never selects an online
recognizer; when allowed, an unavailable recognizer degrades explicitly to
record-only.
- Real-time capture producers use a bounded, non-blocking frame queue.
Overflow is reported as a `CaptureEvent` with the affected track and dropped
PCM frame count (not packet count). The separate control queue is also
bounded and cannot carry sample payloads. Microphone and system audio remain
separate tracks.
- The current Swift `WispAudioKit` remains the macOS compatibility adapter.
Its existing `SessionResult` can be viewed as a backend-neutral
`TranscriptEvent`, avoiding a risky capture rewrite in the first foundation
change.

This boundary is intentionally a foundation, not a claim that every backend is
complete. Linux PipeWire capture, connecting Windows WASAPI frames to actual
local-model inference, and a Nemotron transcriber adapter are follow-up work.

## Requirements

- **macOS 26 (Tahoe)** — Wisp relies on `SpeechAnalyzer`, Core Audio Process Taps, and the new Metal Toolchain, so macOS 26 is required for now.
Expand Down Expand Up @@ -128,7 +156,8 @@ MCP hosts should run the bundled `wisp-mcp` binary over stdio, for example `/App
## Roadmap

- [ ] **Windows support** — WASAPI mic + loopback Ogg/Opus recording is in place; connecting the same PCM stream to local-model transcription is the remaining core path.
- [ ] **Linux support** — exploring PipeWire monitor sources paired with a local Whisper-family model.
- [ ] **Linux support** — implement a PipeWire capture adapter and pair it with a local transcription backend.
- [ ] **Additional local models** — evaluate and implement Nemotron behind `TranscriberBackend`; no Nemotron runtime is bundled today.
- [x] Copy transcript to clipboard (plain text) and export as Markdown (`.md`) with a lightweight, CloudEvents-inspired YAML frontmatter envelope (`id`, `type`, `source`, `time`, `subject`, …).
- [ ] Export to SRT / JSON.
- [ ] Speaker diarization within a single channel.
Expand Down
Loading