⏱️ Game-Time Benchmark

Benchmark for evaluating temporal dynamics in full-duplex spoken dialogue models — time-aware instruction following, tempo, and simultaneous speech.

This repository is a fully open-source implementation of the evaluation pipeline — WhisperX for word-level alignment and Qwen/Qwen3.6-35B-A3B-FP8 as the LLM judge — to improve accessibility and reproducibility. The paper's original pipeline used the commercial Gemini Pro as the judge; everything here runs on open models you can serve yourself. The entire evaluation runs on a single GPU (we used one NVIDIA L40S).

• 🌐 Project page: kwchang.org/Game-Time
• 📄 Paper (ICASSP 2026): arxiv.org/abs/2509.26388
• 🤗 Dataset: gametime-benchmark/gametime
• 🤗 Model outputs: gametime-benchmark/gametime-outputs

📊 Results

Scores (1–5) from the open pipeline (Qwen3.6 judge). We report Instruction Following for most tasks, and Response Appropriateness for the basic open-ended tasks.

📁 Repository layout

eval/alignment/   WhisperX word-level alignment       (self-contained uv project)
eval/llm_judge/   vLLM server + LLM-judge scoring     (self-contained uv project)
examples/         sample audio + alignments + scores for a quick look at the formats
data/             local workspace — downloads land here (not tracked)

The two eval tools have separate environments. Run uv sync inside the tool you need.

📥 Download the benchmark

uvx --from huggingface_hub hf download gametime-benchmark/gametime \
  --repo-type dataset \
  --include "download/*.zip" \
  --local-dir ./data/gametime

for f in data/gametime/download/*.zip; do
  unzip -o "$f" -d data/gametime
done

You'll get basic_instructions/ and advanced_instructions/, each containing audios/ (WAVs), text/ (transcripts), and alignments/ (word-level timestamps).

📥 Download baseline model outputs

Pick one model (or *.zip for all 6):

uvx --from huggingface_hub hf download gametime-benchmark/gametime-outputs \
  --repo-type dataset \
  --include "download/*.zip" \
  --local-dir ./data/gametime-outputs

for f in data/gametime-outputs/download/*.zip; do
  unzip -o "$f" -d data/gametime-outputs
done

Layout mirrors the benchmark: <model>/{basic,advanced}_instructions/audios/<sub-task>/test/<id>.wav. Stereo WAVs with channel 0 = USER, channel 1 = AGENT.

• download/alignments.zip — pre-computed WhisperX word-level alignments for all 6 models, unzipped to <model>/{basic,advanced}_instructions/alignments/.
• download/scores.zip — LLM-judge results for all 6 models, unzipped to <model>/{basic,advanced}_instructions/scores/ (see examples/ for the format).

🎙️ Word-level alignment (WhisperX)

Transcribe dual-channel WAVs into per-file .jsonl alignments (one line per channel: USER first, AGENT second):

cd eval/alignment && uv sync
uv run python annotate.py <wav_dir> <alignment_dir>   # or: bash run_annotate.sh

The script uses whisper large-v3 and runs on cuda (one GPU). Output schema: [word, [start, end], speaker].

⚖️ LLM-judge scoring

Scores each alignment on turn_taking / response_appropriateness / instruction_following (1–5) with Qwen/Qwen3.6-35B-A3B-FP8 served by vLLM.

cd eval/llm_judge && uv sync
bash run_serve.sh                                  # on a GPU node; serves on port 8002
URL=http://<serving-node>:8002/v1 bash run_eval.sh # writes <model>/.../scores/
python print_scores.py                             # paper-style score tables

Already-scored files are skipped, so interrupted runs can simply be re-run.

📚 Citation

@inproceedings{chang2026game,
  title={Game-time: Evaluating temporal dynamics in spoken language models},
  author={Chang, Kai-Wei and Hu, En-Pei and Kuan, Chun-Yi and Ren, Wenze and Chen, Wei-Chih and Lin, Guan-Ting and Tsao, Yu and Sun, Shao-Hua and Lee, Hung-yi and Glass, James},
  booktitle={ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={16302--16306},
  year={2026},
  organization={IEEE}
}