Given a short audio clip from a naturalistic home/lab recording, predict whether the target child is vocalizing. The repo compares ~15 system families — speaker-diarization frontends with ECAPA speaker enrollment, multiple-instance-learning (MIL) classifiers, pseudo-frame self-distillation, encoder baselines, audio LLMs, audio-scene classifiers, and audio-visual fusion — all evaluated on the same splits, and includes a synthetic-scene generator used for data augmentation.
| Document | Contents |
|---|---|
| docs/PIPELINE.md | Every stage in order: input and where it lives, the exact command, what output it produces and where, which environment to run it in. |
| docs/ENVIRONMENTS.md | Every conda/uv/pip environment, how to build it, which scripts need it, required env vars. |
| docs/DATA.md | Input datasets (SAILS BIDS, Providence, Playlogue, TinyVox, LibriSpeech, MUSAN, RIRs), where they live, and the train/val/test split system. |
| docs/RESULTS.md | Catalog of all result directories, the run-directory naming conventions, the per-run file schema, and headline metrics. |
├── whisper-modeling/ USC-SAIL Whisper+LoRA frame classifier (vendored upstream code)
│ + ALL data splits (seen_child_splits/, all_children_splits/, kfold variants)
├── pyannote/ Multi-diarizer evaluation suite (misleading name — covers ALL diarizers):
│ unified.py = shared ECAPA-enrollment harness for 10+ diarization frontends
├── mil/ Multiple-instance-learning classifiers (WavLM/Whisper/HuBERT backbones;
│ frame-window MIL, segment-instance MIL, ACMIL, TS-MIL)
├── pseudo_frame/ Pseudo-frame self-distillation (diarizer-agreement labels → frame head)
│ + audio-visual distillation experiments (spec-015)
├── baselines/ Encoder baselines, audio LLMs (Qwen), scene classifiers (YAMNet/AST),
│ PANNs/CLAP/ECAPA/Parakeet baselines + cross-child splits
├── av_fusion/ Audio-visual late fusion (face detection/tracking, ASD features, GPT-4o)
├── synth/ Synthetic 30s scene generator (v2 canonical corpus; v3 perturb; v4 lit-ext)
├── evaluation/ Metrics, re-tuning, bootstrap CIs, k-fold aggregation, ensembles/routing,
│ frame localization, posthoc per-timepoint tables
├── video/ Video active-speaker-detection bridge (TalkNet/TS-TalkNet/LocoNet/Light-ASD;
│ Python 3.10 env; repos+checkpoints gitignored, see video/SETUP.md)
├── BabAR/ VTC 2.0 child diarizer (BabAR pipeline + standalone VTC)
├── VBx/ VB-HMM diarization
├── encoders/ Encoder feature-extraction helpers
├── specs/ Design docs per feature (spec-001 … spec-022) — the development history
├── docs/ Project documentation (this reorganization)
├── tests/ Tests
│
│ DATA (mostly gitignored, see docs/DATA.md)
├── data/ TinyVox, LibriSpeech, RIRs, MUSAN, extracted segments
├── providence/ Providence corpus audio + RTTM
├── playlogue/ Playlogue corpus audio + RTTM
├── synth_results/ Synthetic corpora (manifests tracked; WAVs gitignored)
│
│ RESULTS (see docs/RESULTS.md for the full catalog and naming conventions)
├── *_ecapa_enrollment_runs*/ one dir per diarizer × split-variant (root level)
├── babar_combined_runs*/, ensemble_runs/, baseline_results*/, models/, logs/
└── ...
conda activate child-vocalizations # docs/ENVIRONMENTS.md §1
cd pyannote
python unified.py --diarizer vtc_kchi # runs VTC-KCHI + ECAPA enrollment on the seen-child split
# → writes vtc_kchi_ecapa_enrollment_runs/ at repo root:
# enroll_val_predictions.csv, enroll_test_predictions.csv, enroll_test_metrics.jsonThe full reproduction order (data → splits → synthetic corpus → training → enrollment → evaluation) is in docs/PIPELINE.md.
whisper-modeling/extends the USC-SAIL public model (github.com/usc-sail/child-adult-diarization).