SyntaxCue — a live-interview desktop app using whisper.cpp for on-device transcription (Metal on macOS, Vulkan on Windows)

[ZetBaur]

Hey all — sharing SyntaxCue, a desktop app (macOS & Windows) that listens to system audio during a live technical interview and streams an AI-generated answer suggestion, using whisper.cpp for fully local transcription (no audio ever leaves the device).

A few whisper.cpp-specific details that might be relevant to this community:

• Bundled default is the Small model, with larger models as optional downloads
• On macOS, model load is ~271ms with Metal
• On Windows, we initially shipped CPU-only by mistake — whisper.cpp was paying a fixed ~25s per transcription pass regardless of utterance length. Wiring up the Vulkan backend for whisper-rs dropped that to ~330ms warm (after an ~8s one-time GPU warm-up), a huge improvement
• Segmentation into utterances is plain RMS-threshold VAD (0.006 threshold, 0.7s silence to close a segment) rather than a neural VAD — kept deliberately simple since a slightly-off split just costs marginally more/less audio, not a wrong split

Full write-up of the pipeline: https://syntaxcue.com/how-syntaxcue-captures-system-audio/

Site: https://syntaxcue.com

Happy to talk more about the Vulkan backend wiring for whisper-rs if useful to anyone else hitting the same CPU-only trap on Windows.