TalkAct — Talking While Acting

A fast/slow dual-model agent that holds a real-time phone conversation while autonomously operating a computer.

Computer-use agents perceive-think-act in multi-second steps, while natural conversation demands responses in under a second — so a single model cannot do both at once. TalkAct decouples the two timescales: a fast conversational model on a cascaded voice pipeline (VAD → ASR → LLM → TTS) holds the call, while a slow computer-use agent drives a browser with Playwright. The two are coupled by a minimal bridge contract:

• a state digest the slow agent attaches to every action (grounds the fast agent's answers at zero extra model calls),
• a bidirectional text channel (@slow: relays from the caller; ask_user / tell_user from the browser agent),
• speech-first streaming and digest-gated completion claims.

On interactive web tasks with a simulated caller holding hidden information, TalkAct matches a single computer-use model on task success while responding ~15× faster (0.6 s vs. 8.6 s median voice latency), inside the human turn-taking band.

This repository contains the system, the VoiceComputerBench benchmark, the evaluation harness, all episode logs, and the paper.

📄 Paper: paper/paper_arxiv.pdf · 🌐 Interactive site: see website/

Repository layout

src/cuv/            core system
  shared.py           SharedState blackboard (digest, transcript, action log, channels)
  fast_agent.py       fast conversational tier (Anthropic / Gemini / local vLLM)
  slow_agent.py       slow computer-use tier (Anthropic tool use + Playwright)
  slow_agent_bu.py    alternate slow backend (browser-use library)
  runner.py           per-episode orchestrator (condition × task)
  simuser.py          LLM-simulated phone caller with a hidden profile
  voice/              cascaded voice pipeline: vad, asr, tts, pipeline
envs/app.py         hermetic Flask task sites (forms / booking / webmail / meeting)
bench/              VoiceComputerBench: tasks, runners, checkers, metrics, plots
results/            raw episode logs (JSON), aggregate tables, and figures
paper/              LaTeX source, figures, and compiled PDF
website/            Vite + React site (architecture, live figures, trajectory replay)
survey/             related-work survey notes
ARCHITECTURE.md · POSITIONING.md · RELATED_WORK.md · REPORT.md

Setup

Requires Python 3.12.

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/playwright install chromium

The models are read through the standard SDK environment variables:

Variable	Used by	Notes
ANTHROPIC_API_KEY	fast + slow tiers	required
GEMINI_API_KEY (or GOOGLE_API_KEY)	simulated caller (default gemini)	required for the default sim user

Optional overrides (defaults in parentheses):

Variable	Purpose
CUV_SLOW_MODEL (claude-opus-4-8)	slow computer-use model
CUV_FAST_MODEL (claude-haiku-4-5)	fast conversational model
CUV_USER_MODEL	force the simulated-caller model
CUV_DIGEST_MODE (standard)	digest ablation mode
CUV_LOCAL_BASE (http://127.0.0.1:8000/v1)	OpenAI-compatible endpoint for a local fast model

Running the benchmark

Start the hermetic task sites (they serve on 127.0.0.1:8321):

.venv/bin/python envs/app.py

Then run one or more conditions over the four tasks:

# primary comparison
.venv/bin/python bench/run_bench.py \
    --tasks forms-insurance booking-flight webmail-report meeting-helper \
    --conditions duplex strawman --seeds 2

Conditions: duplex (ours), strawman (single computer-use model in the loop), sequential (ask-then-act), duplex-blind / duplex-noask (bridge ablations), duplex-bu (browser-use slow backend), fast-only (no browser).

To reproduce the full grid used in the paper:

bash bench/run_grid.sh          # all phases (A–J)
.venv/bin/python bench/analyze.py   # aggregate → results/*.json
.venv/bin/python bench/plots.py     # figures → results/figures/

Episode logs land in results/episodes/*.json. Success is scored against environment ground truth by bench/checkers.py; answer correctness uses a cached LLM judge.

Website

An interactive companion site (animated architecture, live figures from the benchmark JSON, and a replayable trajectory visualizer with re-synthesized audio):

cd website
npm install
npm run dev        # http://localhost:5173
npm run build      # static site in dist/

See website/README.md for the data-regeneration pipeline.

Notes

• The task environments are hermetic self-hosted sites so benchmarks are reproducible and safe (no real reservations, emails, or accounts). All caller profiles are synthetic.
• Some phases evaluate open-weight fast tiers served locally via vLLM (see bench/run_phase_*.sh).
• WebArena integration (a production Magento CMS) uses the standard public demo image; admin/admin1234 is that image's documented default credential, not a real secret.

Citation

If you use this work, please cite the paper (see paper/paper_arxiv.pdf):

@article{talkact2026,
  title  = {Talking While Acting: Real-Time Voice for Slow Computer-Use Agents},
  author = {Li, Bojie and Shi, Noah},
  year   = {2026}
}

License

MIT — see LICENSE.