Scene-aware presence/quality axis + ASR transcription overhaul (CrisperWhisper 2.0, Qwen)

.gitignore

@@ -167,6 +167,10 @@ cython_debug/
 
 # data folder
 data/
+# ...but the audio_analysis workflow ships checked-in package data (AudioSet
+# category map, YAMNet class names) that must be tracked and wheel-included.
+!src/senselab/audio/workflows/audio_analysis/data/
+!src/senselab/audio/workflows/audio_analysis/data/**
 # pdoc documentation
 docs/
 

README.md

@@ -54,6 +54,57 @@ For more detailed information, check out our [**Documentation**](https://sensein
 - **senselab AI**: Interact with your data through an AI-based chatbot. The AI agent generates and runs senselab-based code for you, making exploration easier and giving you both the results and the code used to produce them (perfect for quick experiments or for users who prefer not to code).
 
 
+---
+
+## Adaptive audio analysis (uncertainty-driven)
+
+senselab can analyze a recording with its full task suite (diarization, scene/event tagging, quality
+metrics, multi-model ASR + forced alignment, speaker embeddings), quantify **where** the models are
+uncertain along three temporal axes — *presence* (is someone speaking?), *identity* (who?), and
+*utterance* (what was said?) — and then **act on that uncertainty**: a deterministic, budgeted loop
+re-processes only the uncertain regions (extra ASR models, embedding re-clustering, overlap
+detection), fuses a consensus transcript/diarization, and explains any residual uncertainty instead
+of hiding it. Design + results: `specs/20260723-225523-dynamic-uncertainty-workflow/`.
+
+It runs in two steps:
+
+```bash
+# Step 1 — analyze: run every model on the recording (results are content-addressably
+# cached, so re-runs are cheap). `--enhancement auto` adds a triage round 0 that skips
+# the speech-enhancement pass on clean audio and stops early on silent recordings.
+uv run python scripts/analyze_audio.py path/to/recording.wav --enhancement auto
+# → artifacts/analyze_audio/<name>_<timestamp>/  (per-task JSONs, 9 uncertainty
+#   parquets, Label Studio bundle, disagreements.json, timeline.png)
+
+# Step 2 — adapt: run the uncertainty-driven loop over that run directory.
+uv run python scripts/adaptive_loop.py artifacts/analyze_audio/<run_dir> \
+    --cache-dir artifacts/analyze_audio_cache \
+    --ground-truth path/to/labelstudio_export.json   # optional: scores vs human labels
+```
+
+The loop writes, under the run directory (or `--out`):
+
+- `final/transcript.json` — consensus word-level transcript (family-weighted voting across all ASR
+  models) with speaker attribution, per-word confidence, and alternates where models disagree;
+- `final/diarization.json` + `final/presence.parquet` — refined speaker segments (embedding
+  change-point + re-clustering repair) and the fused speech-presence track;
+- `final/convergence.json` + `final/iterations.json` — what the loop did and why: every intervention
+  (fired / deferred / blocked) with trigger values and measured uncertainty deltas, budget
+  accounting, and regions marked `converged` / `irreducible` (with a machine-readable reason);
+- `final/timeline.png` — ground truth (if given) vs presence / identity / utterance uncertainty per
+  round, interventions, and the confidence-colored fused words;
+- `final/labelstudio_{tasks,config}.{json,xml}` + `disagreements_resolved.json` — the original Label
+  Studio bundle with `final__*` consensus tracks added, and the round-1 disagreements annotated with
+  their resolutions.
+
+Useful knobs (all thresholds/budgets/models live in a versioned policy file —
+`src/senselab/audio/workflows/audio_analysis/adaptive/policy/default.yaml` — overridable via
+`--policy my_policy.yaml`): `--max-rounds`, `--aggregator`, per-run intervention budgets, ASR
+reserve/escalation pools, and identity-repair parameters. Runs are **deterministic**: identical
+inputs + policy produce byte-identical decision logs. `HF_TOKEN` enables the gated
+`pyannote/segmentation-3.0` overlap detector; without it the loop degrades gracefully and records
+the skipped intervention in `convergence.json → next_actions`.
+
 ---
 
 ## ⚠️ System Requirements

pyproject.toml

@@ -208,4 +208,4 @@ skip = [
   "*.cha",
   "*.ipynb"
 ]
-ignore-words-list = ["senselab", "nd", "astroid", "wil", "SER", "te", "EXPRESSO", "VAI", "vie", "g2p"]
+ignore-words-list = ["senselab", "nd", "astroid", "wil", "SER", "te", "EXPRESSO", "VAI", "vie", "g2p", "pres", "befores", "trough", "bellow", "coo", "patter", "AER"]

scripts/adaptive_loop.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+"""Adaptive uncertainty loop (prototype driver).
+
+Runs the loop from ``specs/20260723-225523-dynamic-uncertainty-workflow`` over a
+completed ``analyze_audio.py`` run directory: round 1 ingests the run's
+uncertainty parquets into the belief store (with a re-aggregation parity
+check), rounds 2..K execute the policy-ranked intervention catalog (stream
+election, hallucination adjudication, missed-speech correction, cache-replay
+reserve-ASR escalation; live-backend rules defer with guard reasons), and the
+final round fuses a consensus transcript / diarization / presence track with a
+full decision audit trail.
+
+Usage:
+    uv run python scripts/adaptive_loop.py artifacts/e2e_runs/<run_dir> \
+        --cache-dir artifacts/analyze_audio_cache \
+        --ground-truth ~/Downloads/updated-label-XXXX.json
+
+Works in minimal environments (no torch): heavy senselab imports are bypassed
+via submodule loading when the full package cannot import.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+_REPO_SRC = Path(__file__).resolve().parents[1] / "src"
+
+
+def _ensure_light_importable() -> None:
+    """Make senselab importable when running from a checkout without installation.
+
+    The package ``__init__`` chain is lazy (PEP 562 — see T046 in
+    ``specs/20260723-225523-dynamic-uncertainty-workflow/architecture-review.md``),
+    so the loop's pure submodules import without torch/transformers; all that is
+    needed here is the src/ path when senselab isn't pip-installed.
+    """
+    try:
+        import senselab.audio.workflows.audio_analysis.aggregate  # noqa: F401
+    except ImportError:
+        if str(_REPO_SRC) not in sys.path:
+            sys.path.insert(0, str(_REPO_SRC))
+
+
+def main(argv: list[str] | None = None) -> int:
+    """CLI entry point."""
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("run_dir", type=Path, help="Completed analyze_audio run directory")
+    parser.add_argument("--cache-dir", type=Path, default=Path("artifacts/analyze_audio_cache"))
+    parser.add_argument("--policy", type=Path, default=None, help="Policy YAML overriding the default")
+    parser.add_argument("--out", type=Path, default=None, help="Output dir (default: run_dir)")
+    parser.add_argument("--max-rounds", type=int, default=3)
+    parser.add_argument("--aggregator", default=None, help="Override (default: from run's disagreements.json)")
+    parser.add_argument("--ground-truth", type=Path, default=None, help="Label Studio export JSON")
+    args = parser.parse_args(argv)
+
+    if not (args.run_dir / "summary.json").exists():
+        print(f"ERROR: {args.run_dir} is not an analyze_audio run dir (no summary.json)", file=sys.stderr)
+        return 2
+
+    _ensure_light_importable()
+    from senselab.audio.workflows.audio_analysis.adaptive.evaluate import evaluate_against_ground_truth
+    from senselab.audio.workflows.audio_analysis.adaptive.loop import run_adaptive_loop
+
+    result = run_adaptive_loop(
+        args.run_dir,
+        cache_dir=args.cache_dir,
+        policy_path=args.policy,
+        out_dir=args.out,
+        max_rounds=args.max_rounds,
+        aggregator=args.aggregator,
+    )
+    out_dir = Path(result["out_dir"])
+    print(f"run_state: {result['run_state']}  rounds: {result['rounds']}")
+    print("parity:", json.dumps(result["parity_check"]))
+    print(
+        f"interventions fired: {result['n_interventions_fired']}  "
+        f"fused words: {result['n_words_fused']}  stream: {result['fusion_stream']}"
+    )
+    print(f"final/: {out_dir / 'final'}")
+
+    # Visual timeline — best-effort sidecar (mirrors analyze_audio's timeline.png).
+    try:
+        from senselab.audio.workflows.audio_analysis.adaptive.plot import build_adaptive_timeline
+
+        plot_path = build_adaptive_timeline(out_dir, gt_path=args.ground_truth, title=args.run_dir.name)
+        if plot_path is not None:
+            print(f"timeline: {plot_path}")
+    except Exception as exc:  # noqa: BLE001 — plotting must never fail the run
+        print(f"warn: timeline plot failed: {exc!r}", file=sys.stderr)
+
+    if args.ground_truth is not None:
+        eval_doc = evaluate_against_ground_truth(
+            out_dir=out_dir, gt_path=args.ground_truth, word_streams=result["word_streams"]
+        )
+        t = eval_doc["transcript"]
+        print("\n── evaluation vs ground truth ──")
+        print(f"presence: {json.dumps(eval_doc['presence'])}")
+        print(f"fused WER: {t['fused']['wer']}  (normalized: {t['fused']['wer_normalized']})")
+        for m, s in (t.get("per_model") or {}).items():
+            print(f"  {m}: WER {s['wer']} (normalized {s['wer_normalized']})")
+        print(f"diarization: {json.dumps(eval_doc['diarization'])}")
+        print(f"localization: {json.dumps(eval_doc['localization'])}")
+        print(f"eval.json: {out_dir / 'final' / 'eval.json'}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())