Scene-aware presence/quality axis + ASR transcription overhaul (CrisperWhisper 2.0, Qwen) - #536
Merged
Merged
Conversation
Rework the presence axis into a scene-aware dimension that also quantifies audio quality (SNR, clipping, reverb, bandwidth) and background sound sources (people/machine/environment), add per-axis temporal grids + frame-level posteriors, and improve the utterance uncertainty estimator. Presence axis name and aggregated_uncertainty output are preserved; all new signals are additive columns. Reuses existing quality_control metrics + SQUIM and adds pyannote/brouhaha (gated, no new dep, no subprocess venv) for frame-level VAD + SNR + C50. Calibration bootstrapped from synthetic clean+noise+RIR mixtures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 0 (research.md, D1-D10) + Phase 1 (plan.md, data-model.md,
contracts/, quickstart.md) for the spec-kit workflow.
Key decisions grounded in codebase research:
- Frame posteriors via pyannote segmentation-3.0 + brouhaha (Model+Inference,
ensure_hf_model/local_files_only); no new dep, no subprocess venv.
- Quality: reuse quality_control DSP (clip/SNR) + brouhaha (SNR/C50) +
librosa rolloff (bandwidth); computed on a 0.5s analysis window broadcast
to the finer presence grid.
- Sources: raise scene top_k to full distributions + hand-authored AudioSet
display-name -> {speech,people,machine,environment} map (no ontology in repo).
- Presence already uses mean-p_voice (not Shannon entropy); surface
confidence/uncertainty split as additive columns.
- Utterance token entropy requires plumbing Whisper output_scores and
extending ScriptLine (also revives the dead avg_logprob signal) - highest
risk, phase 4, splittable.
- Calibration from synthetic noise + decay-RIR (numpy, no new dep).
Constitution: PASS with two justified deviations (backward-compat additive
columns; ScriptLine optional fields) recorded in Complexity Tracking.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
46 tasks across 8 phases (setup, foundational, US1-US5, polish), organized by user story for independent delivery. Tests-first per story. Records the US4 (ScriptLine/Whisper token) risk containment and the compute.py shared integration-point serialization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…1-T005) Phase 1-2 (setup + foundational) of the scene-quality/utterance rework: - librosa promoted to an explicit dependency (spectral rolloff for bandwidth) - UncertaintyRow gains 14 defaulted scene/quality/source/utterance fields - write_axis_parquet emits matching nullable columns (uniform schema across the three axis parquets) - scene_quality package scaffold Existing workflow readers unaffected (22 regression tests pass; aggregated_uncertainty unchanged). Foundation for US1-US5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (T006-T011) Adds four [0,1] audio-quality degradation scores + quality_uncertainty per presence bucket: - quality_snr ← Brouhaha frame SNR (primary) x-checked vs existing DSP SNR - quality_clip ← existing proportion_clipped_metric - quality_reverb ← Brouhaha C50 - quality_bandwidth ← librosa spectral rolloff vs Nyquist - quality_uncertainty ← spread among independent SNR estimators New pyannote/brouhaha loader (Model+Inference, ensure_hf_model/local_files_only, gated HF token; no new dep, no subprocess venv). Quality computed on a fixed 0.5s/0.25s analysis window and broadcast onto presence buckets (analysis resolution != reporting grid). Null-safe when the model/audio is unavailable (FR-023); silent buckets report null signal-dependent quality. compute_uncertainty_axes gains scene_quality=True; provenance records the analysis window + brouhaha id/revision. Existing hermetic tests stub the loader offline. 8 model-free quality tests + 2 end-to-end wiring tests. mypy/ruff/ codespell clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…012-T017) Adds per-bucket AudioSet source-category masses to the presence axis: src_speech / src_people / src_machine / src_environment + src_dominant. - data/audioset_source_map.json: versioned map of all 527 AST classes to the 4 categories, derived by walking each class to its top-level AudioSet ontology ancestor (Speech-subtree->speech, Human sounds->people, Sounds of things->machine, Music/Animal/Natural/Source-ambiguous/Channel-env-> environment). YAMNet's 521 classes are a complete subset (vendored list + coverage test, SC-003). - sound_sources.py harvester: sums per-window scores into category masses, averages AST+YAMNet, normalizes, picks dominant; nearest-window projection. - compute.py: sound_sources=True wires src_* columns + provenance additively. - scripts/analyze_audio.py: --scene-top-k (default 50) persists enough of the distribution for meaningful masses; top-1 consumers unaffected. - .gitignore: negate the broad data/ ignore for this package's shipped data. Null-safe when neither classifier ran (FR-023). 5 source tests + 1 wiring test; 95 workflow tests green; ruff/mypy/codespell clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…18-T025) - frame_posteriors.py: continuous per-frame P(speech) from pyannote segmentation-3.0 raw scores (Model+Inference, ensure_hf_model/local_files_only, gated; powerset -> 1-P(silence)); FramePosterior.mean_std_in_window. - Per-axis grids: compute_uncertainty_axes gains presence_grid (default None -> shared grid, so legacy callers unchanged); CLI --presence-grid-win/hop default 0.1/0.02. Quality + sources now computed on the presence grid. - presence.py: two frame-posterior voters (segmentation-3.0 + Brouhaha VAD); coarse voters (AST/YAMNet/no_speech_prob/ASR-overlap/embedding) demoted to weight 0.25 on grids finer than 0.5s (FR-014) via a weighted p_voice mean. - aggregate.py: weighted _weighted_p_voice shared by aggregate_presence and presence_p_voice; weights default 1.0 so existing values are unchanged. - Confidence/uncertainty split columns: presence_confidence (= p_voice) and presence_uncertainty (decisiveness OR within-bucket frame instability). aggregated_uncertainty stays byte-for-byte the legacy value (SC-008). - Intensity mask unified on overlap-averaging so it's correct when presence runs on a finer grid than identity/utterance. frame_posteriors + grid tests; split-column + instability test. 82 workflow tests pass (excl. unrelated PII); ruff/mypy/codespell clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaced by running the gated models with a real HF_TOKEN: - Null-safety (FR-023): PyannoteAudioModel validates against HF *at construction* and raises pydantic ValidationError for an inaccessible gated repo — that was OUTSIDE the try/except, crashing the workflow. Moved model construction inside the guard in both loaders (+ regression tests). - Frame resolution: default pyannote Inference returns a coarse ~1s per-chunk aggregate; segmentation-3.0 now uses window="whole" for the native ~17ms per-frame posterior (validated: 0.0169s hop). Frame hop derived from the model receptive_field (window="whole" returns a bare ndarray, not a SlidingWindowFeature). Long-file sliding+stitching noted as follow-up. - quality_bandwidth reimplemented on torch.stft instead of librosa (prefer torch for new algorithms); reverted the librosa pyproject/uv.lock addition (librosa was already transitive + directly used by quality_control). - Script: scene quality is a REQUIRED signal — analyze_audio.py now hard-fails (exit 2) if Brouhaha is unavailable rather than silently shipping null quality columns; added --no-scene-quality / --no-sound-sources opt-outs. Known blocker (surfaced, not yet resolved): pyannote/brouhaha is not a plain pyannote-audio model — its checkpoint needs the `brouhaha` module from the GitHub-only brouhaha-vad package, which pins old pyannote incompatible with our 4.0.4. Loading path (subprocess venv vs DSP fallback) pending decision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the single window="whole" pass (which pyannote warns risks memory / degraded results on long, frame-based inputs) with chunked_frame_inference: slides bounded 10s windows (2s overlap), runs a whole-window pass per chunk, maps chunk-local frames to absolute frame indices, and overlap-averages into one continuous per-frame timeline. Flat memory for any length; native ~17ms resolution preserved. Short clips (<=10s) stay a single pass. Shared by both the segmentation-3.0 speech-posterior extractor and the Brouhaha loader (dedup of the earlier per-module output/frame-grid helpers). Validated on the real segmentation-3.0 model: a 15s input yields frame-level output spanning the full duration at 0.0169s hop across multiple stitched chunks. Unit tests cover the single-pass and stitched paths with a fake inference. 86 workflow tests pass; ruff/mypy/codespell clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pyannote/brouhaha isn't loadable in the main env — its model class lives in the
GitHub-only brouhaha-vad package, which pins pyannote.audio 3.1-3.3 /
speechbrain<1.0 / numpy<2.0 (incompatible with our pyannote 4.0.4). So Brouhaha
now runs in a dedicated venv (same pattern as the NeMo/Qwen ASR backends):
- ensure_venv("brouhaha", [brouhaha @ git, torch<2.3, numpy<2, soundfile]);
torch routed via the CUDA-aware index by ensure_venv.
- Parent chunks each audio with senselab's Audio.window_generator (overlapping
~10s windows), saves chunk wavs; worker loads the gated model (HF_TOKEN from
env), runs whole-window Inference per chunk, saves per-frame (VAD,SNR,C50)
npy; parent stitches via the shared stitch_frames helper into BrouhahaFrames.
- Null-safe (FR-023): venv-build or worker failure → [None]; analyze_audio.py
still hard-fails on all-null (scene quality is required there).
frame_posteriors: extracted the pure stitch_frames helper (shared by the
in-process segmentation extractor and the Brouhaha subprocess path).
Tests: venv-unavailable null-safety, mocked-worker frame assembly, stitch_frames
unit coverage. 87 workflow tests pass; ruff/mypy/codespell clean. Real venv
build validation pending (old-stack pins may need tuning on first build).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
brouhaha-vad pins pyannote 3.x but not huggingface_hub; uv pulled a hub too new (dropped the use_auth_token kwarg pyannote 3.x passes to hf_hub_download), failing the worker load. Pin huggingface_hub>=0.19,<0.24. Validated end-to-end against the real gated model (subprocess venv builds, worker loads pyannote/brouhaha, runs inference): 288 frames @ 0.0169s on the test clip; VAD in [0,1], SNR ~12dB, C50 ~27dB — confirming the channel order and dB denormalization. Brouhaha SNR/C50 quality signals now functional. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…FR-024) Adds two rows to the aligned timeline PNG, sourced from the presence axis's additive columns and rendered only when present: - scene quality: the four [0,1] degradation lines (snr/clip/reverb/bandwidth), raw solid + enhanced dashed; - sound sources: stacked category masses (speech/people/machine/environment) for the raw pass. Refactors the row-index map to a running counter so inserting rows no longer needs fragile hardcoded offsets. Existing rows/behavior unchanged when the new columns are absent (scene_quality/sound_sources disabled). Test + synthetic render confirm the new rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
qwen-asr depends on librosa without a version floor; in a fresh Python 3.12
venv uv backtracks librosa -> numba -> llvmlite to llvmlite 0.36.0, which
predates 3.12 support, failing the install ("Cannot install on Python version
3.12.0"). Pinning numba>=0.60 keeps the chain on llvmlite>=0.43 (numba 0.66 /
llvmlite 0.48 / librosa 0.11 — all 3.12-compatible). Verified by clean-venv
dry-run resolution. Unblocks the Qwen3-ASR backend in analyze_audio runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified against a Label Studio ground-truth annotation of the 4-speaker audio_48khz_mono_16bits clip (brief 0.38s turn, ~0.4s gap, background noise). Generic changes (not clip-specific): - quality_bandwidth: raise the rolloff percentile 0.85 -> 0.95 so it tracks the upper spectral edge (band-limiting / telephone) instead of the voiced-energy tilt. An 85% rolloff sat at ~1.8 kHz even for full-band speech, flagging every recording as ~0.77 band-limited; at 0.95 the same clip reads ~0.27-0.38 and a real 3.4 kHz telephone cut still reads ~0.57. - Identity windows/hops: --cross-stream-win/hop 0.5 -> 0.25 and --embedding-hop-s 0.5 -> 0.25 so sub-second speaker turns localize (kept the 1s ECAPA window). Embedding-clustered speakers went from 9 -> 17 windows with boundaries that track the GT turn structure. Fixed the stale --cross-stream help (presence has its own grid now). Presence left as-is: recall 1.0 vs GT, resolves the brief turn, and the ~0.4s GT "gap" is correctly high-uncertainty (both trained VADs detect voice-like activity there — a real model/annotation disagreement, not a bug). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Transcription-cleanup overhaul (part 1 of the ASR rework): - New crisperwhisper.py subprocess-venv backend for CrisperWhisper 2.0 (verbatim, word-level timestamps). Platform-aware backend: ct2 on Linux x86_64 (ctranslate2-crisperwhisper wheels are linux-x86-only), transformers backend elsewhere (macOS arm64 dev) loading the same safetensors weights. Standard ctranslate2 pinned on the transformers path to satisfy the library's unconditional top-level `import ctranslate2` (engine/hallucination modules). Native per-word/line confidence flows via ScriptLine.score when the library exposes it (2.0.1's WordTimestamp does not, so it degrades to None). - api.py: route nyralabs/CrisperWhisper2.0* -> CrisperWhisperASR. - analyze_audio.py default ASR set: CrisperWhisper2.0_turbo + canary-qwen-2.5b + Qwen3-ASR-1.7B (whisper-large-v3-turbo + granite dropped from defaults, kept as opt-in). Docstring updated. Validated end-to-end on the GT clip (transformers backend): correct verbatim text with word timestamps tracking the ground-truth turn boundaries, including the brief 0.38s "Kenny" turn. Hermetic assembly + platform tests + a venv-present integration smoke test. ruff/mypy/codespell clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ded) Narrow the SNR->degradation window to 25 dB (clean) .. 5 dB (floor) so quality discriminates noisy from clean recordings: clean conversational/studio speech (>=25 dB) -> ~0, a noisy recording (~10-15 dB) -> 0.5-0.75, heavy noise (<=5 dB) -> 1.0. The GT-annotated noisy clip's ~12 dB Brouhaha SNR now reads 0.65 (was 0.60 at the old 30..0 span), clearly elevated vs a clean recording's ~0. Generic, not clip-specific; overridable via the calibration profile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d ceiling The smoke test called ensure_venv WITHOUT max_cuda_version, so it reproduced the old cu128 routing regardless of the fix (which lives in extract_brouhaha_frames) — failing on the GPU runner even though the fix is correct. Pass _BROUHAHA_MAX_CUDA_VERSION (the production constant) so the test builds through cu121 like production does, and assert the ceiling is declared. This also tightens the guard: blank/remove the ceiling and the build routes cu128 and fails. Only this test failed on the prior GPU run (785 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
satra
removed
ec2-gpu-test
Collaborator
Author
|
Thanks @wilke0818 — the diagnosis is exactly right, and confirmed against the code + reproduced. Reproduced directly with
Fix (your first suggestion — cap to the newest compatible index):
Validation: unit coverage for the ceiling in Commits: fix On the secondary blocker — you're right that |
…ask APIs Architecture-review follow-up (T049/T050): pull the transcript-fusion and forced-alignment math out of the adaptive workflow into reusable task packages, and consolidate the duplicated ScriptLine tree-walks. - New `senselab.audio.tasks.speech_to_text_ensemble` (fuse_word_streams, iter_word_leaves, load_calibrator) — the ROVER-lite word-slot voting, now workflow-agnostic. adaptive/fusion.py re-exports these for loop callers and keeps only the workflow-specific parts (word-stream collection, policy→weights translation, belief-state lookups, final/ artifact writers). - New `forced_alignment/mms_fa.py::align_words_mms_fa` — extracted MMS aligner. - `ScriptLine.iter_leaves()` consolidates the recursive word-leaf walk that was re-implemented across forced_alignment, plotting, and the workflow. - Threads through backends/audio_io/evaluate/interventions/loop, aggregate, labelstudio, frame_posteriors, and the evaluation utils accordingly. Fixes the `w` weight/word-dict shadowing (float vs dict[str,Any]) reintroduced in the extracted fuse_word_streams. Full pre-commit (ruff/mypy/codespell/json) green repo-wide; 245 affected tests pass locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng (US4)
Completes User Story 4 of the scene-quality/utterance spec (T026-T034):
the utterance axis now hears a single model's private doubt, not just
cross-model transcript disagreement.
- ScriptLine gains optional avg_logprob / no_speech_prob / token_entropy
(T028), mapped in from_dict. None for every non-Whisper backend, which
revives the previously-dead avg_logprob reads in the harvesters.
- New speech_to_text/token_confidence.py (T029) extracts per-token softmax
entropy, avg_logprob and Whisper's no_speech_prob.
- harvesters.mean_token_entropy_in_window + a token_entropy vote (T030),
folded into aggregate_utterance as a fourth sub-signal (T031) through the
existing --uncertainty-aggregator.
- ASR confidences are temperature-scaled onto a common [0,1] scale (T032);
T=1.0 default reproduces the historical numbers bit-for-bit.
- votes.scene_quality_coupling (T033) raises reported utterance uncertainty
where SNR is poor or a machine/environment source competes, recorded on
the scene_quality_coupling column with the pre-coupling value kept on
raw_aggregated_uncertainty and in model_votes — visible, not invisible
(FR-019). New --utterance-scene-coupling-weights flag (T034).
Two deviations from the contract, both forced by transformers 5.5.4 and
documented in tasks.md + the module docstring:
- Contract A's `pipe(generate_kwargs={"output_scores": True})` cannot work:
AutomaticSpeechRecognitionPipeline._forward keeps only sequences and
token_timestamps and drops the scores before postprocess. Implemented as
a context manager that temporarily wraps pipe.model.generate to observe
its output, leaving pipeline behavior untouched.
- Requests output_logits (raw) rather than output_scores (post-processor),
since Whisper's logits processors suppress <|nospeech|> to -inf, which
would pin no_speech_prob at 0 and distort the entropy.
Verified end-to-end on openai/whisper-tiny, not just in unit tests: the
first cut returned all-None because Whisper's long-form path nests each
generate result under segments[batch][window]["result"] with unbatched 1-D
tensors, and because whisper-tiny spells the token <|nocaptions|> while
<|nospeech|> maps to the unk id. Both are now handled and covered by tests.
A real run yields avg_logprob=-0.61, no_speech_prob=1.8e-08, and 175 token
entropies spanning 0.02-4.29 nats (mean 1.25) — the range that motivated
the documented 3.0-nat normalization reference over log(vocab).
Behavior change: utterance aggregated_uncertainty now carries the coupling
multiplier when scene columns are present. Per SC-008 values are unchanged
when the scene features are disabled (absent columns give coupling 1.0);
the existing votes_test assertion was updated to state this explicitly.
234 passed / 8 skipped across the audio_analysis, speech_to_text and
data_structures suites; ruff + mypy clean on all touched files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the 3 cpu-tests failures from 581b18b. Both causes are in the capture seam, not the uncertainty math. 1. `'ModelOutput' object has no attribute 'dtype'` (2 failures). The seam injected return_dict_in_generate=True unconditionally, which changes what generate *returns*. Whisper's word-timestamp path already requests dict output so it was unaffected, but plain greedy decoding expects a tensor and broke downstream. The wrapper now restores the caller's return type — handing back `sequences` unless the caller itself asked for a dict — so the claim that observing the decoder leaves pipeline behavior untouched is actually true. A backend whose generate rejects the kwargs is re-run clean. 2. `TypeError: 'Mock' object is not iterable` (1 failure). whisper_token_ids did `set(tokenizer.all_special_ids)`, which raises on a Mock tokenizer. Non-int and non-iterable values are now filtered, since failing a whole transcription over a purely additive signal is the wrong trade. Root cause of missing these locally: I ran the `speech_to_text/` directory and never the sibling `speech_to_text_test.py` file, which holds the mock-pipeline and parametrized HF-model tests. Both are now in the verified set: 51 passed across both paths, 29 passed in token_confidence_test, and a real whisper-tiny run still yields avg_logprob=-0.61, no_speech_prob=1.8e-08 and 175 token entropies (mean 1.25 nats) — unchanged by the fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… compat) The scene->utterance coupling (581b18b) multiplies the reported utterance aggregated_uncertainty by the scene multiplier, which is not a pure function of the votes. The adaptive belief store now compares its re-aggregation against raw_aggregated_uncertainty (pre-coupling; identical on pre-coupling artifacts) and documents the scale in disagreements_resolved.json. Verified: raw-anchored parity ALL CLEAN on 538/538 buckets of the reference run; 23/23 votes+adaptive unit tests.
…eements (FR-024, T040/T042) Makes the scene columns visible to the two human-facing consumers, additively. - Label Studio gains `<pass>__presence__quality` and `<pass>__presence__sources` tracks, emitted only when the underlying columns exist — so a run with the scene features disabled produces a byte-identical bundle to before. - disagreements entries now carry presence_uncertainty / quality_uncertainty / src_dominant, with an equal-aggregated tiebreak on presence_uncertainty. The primary ranking is untouched, keeping SC-008 intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tive schema completion (T035-T039, T042/T044/T045) US5 calibration (T035-T039). data-model §5 specifies a nested versioned profile while quality.py / aggregate.py consume flat keys, so calibration.py is the bridge: load_calibration_profile (bundled → documented-defaults fallback, fail-loud validation) plus profile_to_runtime. scripts/calibrate_scene_quality.py synthesizes white/pink-noise SNR sweeps and exponential-RIR RT60 sweeps, runs the workflow's own estimators, least-squares-fits the dB anchors, and emits the FR-022 validation plot + table behind a monotonicity gate (SC-007). SNR is always fitted; C50 only when Brouhaha is available, since the DSP path has no reverb estimator — the profile provenance records which. --calibration-profile threads the runtime dict into both consumers via compute_uncertainty_axes(calibration=...) and comparator_params. Known gap, recorded in tasks.md: per-axis temperatures are CLI passthroughs. Fitting them needs labeled correctness (the adaptive eval harness), so that US5 sub-item is left open by design rather than faked with a placeholder. Adaptive schema completion. from_harvests gains its integration test (T044); final/diarization gains the RTTM sidecar, member_labels and per-segment overlap flags, and the transcript a language field (T042); contracts/interventions.md carries an implementation-alignment section (T045). Local CLAUDE.md was updated for the new flags but is deliberately left untracked, matching how this repo already treats it and AGENTS.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… bug `uv run mypy .` reported 15 errors across 7 files; the repo now type-checks clean (279 files, 0 errors). These were pre-existing — CI stayed green because pre-commit's mypy hook only sees files changed in a commit. One was a real runtime defect, not an annotation nit: - `dependencies.py::_cached_error` constructed `RepositoryNotFoundError(msg)` / `RevisionNotFoundError(msg)`, but huggingface_hub's `HfHubHTTPError` takes `response` as a *required* keyword-only argument. Replaying a cached negative model lookup therefore raised `TypeError` instead of the repo/revision error callers catch. Now passes a synthetic 404 response — with its `request` set, since `HfHubHTTPError` reads `response.request` while building the message and httpx raises `RuntimeError` otherwise. Verified both types construct, raise and catch correctly. - `sparc.py` declared `-> List[Dict[str, torch.Tensor]]` while returning a `ft_len` scalar among the tensors; the annotation was simply untrue, so it is now `List[Dict[str, Any]]` with the reason documented. No behavior change. Remaining were genuine narrowing gaps: `language.py` dereferenced a possibly `None` pycountry lookup (unreachable given the field validator — now asserted explicitly, preserving the documented AttributeError for languages with no ISO 639-1 code); `talk_bank_helpers.py` iterated `tokens` which is optional; `metrics.py` rebound Tensor-typed names to numpy views (split into separate names so the static type matches what librosa/scipy receive); `embeddings_extraction` cast AutoTokenizer's backend union to the cached interface; `frame_posteriors_test.py` marked its duck-typed Inference fake. Also added "AER" (Alignment Error Rate) to `[tool.codespell]` ignore-words, so the `uv run codespell` command documented in CLAUDE.md matches what pre-commit already whitelists via its args. Verified: mypy clean, ruff check + format clean, codespell clean, and 485 passed / 9 skipped across the quality_control, utils, text and audio_analysis suites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…037 run)
Ran scripts/calibrate_scene_quality.py with Brouhaha available, so both the SNR
and C50 anchors are measured rather than assumed. SC-007 monotonicity gate
passed. The profile now ships at
workflows/audio_analysis/data/scene_quality_calibration.json and is the default
— a run without --calibration-profile picks it up.
The fit corrects a real systematic bias, which is the point of US5 rather than
just another artifact. Brouhaha's SNR estimate is strongly compressed (fitted
slope 0.41): a true 0-30 dB sweep spans only ≈4.7-16.6 dB of estimator output.
The uncalibrated anchors (clean 25 dB / floor 5 dB) assumed a true-dB scale, so:
true SNR Brouhaha est default degradation fitted
30 dB 16.57 0.421 0.000
25 dB 15.92 0.454 0.000
20 dB 14.58 0.521 0.141
10 dB 9.94 0.753 0.703
0 dB 4.74 1.000 1.000
A pristine 30 dB clip was reported as 42% degraded, with the whole scale
squashed into [0.42, 1.0] — almost no dynamic range across the clean half. That
also fed the FR-019 coupling I added in 581b18b, inflating utterance
uncertainty on clean speech by ≈1.21x for no reason. Fitted anchors
(clean 15.75 / floor 7.50, in estimator space) map that clip to 0.0.
Test fix: test_default_profile_when_no_path_and_no_bundle asserted the
defaults-fallback but never isolated the bundle — it passed only because no
profile shipped. It now redirects BUNDLED_PROFILE_PATH via monkeypatch so it
tests the fallback it names, and two new tests cover what became the real
default path (bundled profile preferred; it validates and maps to the flat
runtime keys).
Also fixed a stale docstring: _bandwidth_degradation said "85% spectral
rolloff" while _DEFAULT_BANDWIDTH_ROLLOFF_PCT has been 0.95 since the
quality_bandwidth tuning commit. Note the profile's `bandwidth` block is
currently inert — quality.py reads the module constant, not the profile.
Scene spec tasks.md is now at zero unchecked items (T045/T046 executed here:
ruff + mypy clean at 279 files, 162 passed / 3 skipped in the workflow suite).
The one US5 sub-item still open by design is per-axis temperature fitting,
which needs labeled correctness from the adaptive eval harness.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI pre-commit failed on 88e812f: the bundled profile was written with indent=2 and no trailing newline, which end-of-file-fixer and pretty-format-json (--indent=4 --no-sort-keys) both rewrite. Fixed at the source rather than just reformatting the artifact — the fitter now writes indent=4 plus a trailing newline for both the profile and the validation table, so a future run that targets the package tree is committable without a reformat round-trip. Reformatting is whitespace only: the bundled profile is byte-for-byte content-identical to the fitter's output (verified by json comparison), and the loaded anchors are unchanged at clean 15.747 / floor 7.495. `pre-commit run --all-files` now passes with zero failing hooks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wilke0818
added a commit
that referenced
this pull request
Jul 28, 2026
#536 (scene-quality/utterance) + #538 (cached-inference refactor) merged to alpha, so their model-load sites now live here too. Route them through the helpers (cherry-picked from the hf-cache-on-536 integration branch), guard-driven: In-process (SHA-pin / snapshot-path): - voice_activity_detection/frame_posteriors.py (use the SHA ensure_hf_model returns) - workflows/audio_analysis/adaptive/audio_io.py (SepFormer via snapshot path) - workflows/audio_analysis/adaptive/backends.py (ASR pipeline via load_hf_resilient) Subprocess (hf_subprocess_env): - scene_quality/brouhaha.py (gated; token forwarded, online fallback preserved) - speech_to_text/crisperwhisper.py (online fallback preserved) Guard-test allowlist updated to cover all five; PR #1 now covers every HF load site on alpha. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 28, 2026
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks the
audio_analysisuncertainty workflow's presence axis into a scene-aware dimension and overhauls the ASR / transcription stack, driven bySPEECH_PRESENCE_CERTAINTY_ANALYSIS.md, a spec-kit design (specs/20260722-175022-scene-quality-utterance/), and validation against a Label Studio ground-truth annotation of the 4-speaker, noisyaudio_48khz_mono_16bitsclip.All new presence signals are additive columns — the
presenceaxis name, parquet paths, LS tracks, andaggregated_uncertaintyare unchanged, so downstream consumers keep working.Scene-aware presence axis
[0,1]degradation):quality_snr(Brouhaha frame SNR ✕ existing DSP SNR),quality_clip(existingproportion_clipped_metric),quality_reverb(Brouhaha C50),quality_bandwidth(torch.stft spectral rolloff vs Nyquist), plusquality_uncertainty(estimator spread).src_{speech,people,machine,environment}+src_dominant, from a checked-in AudioSet→category map over full AST/YAMNet distributions (--scene-top-k).pyannote/segmentation-3.0(+ Brouhaha VAD), per-axis grids (presence 0.1 s/0.02 s), and apresence_confidence/presence_uncertaintysplit; coarse voters demoted on fine grids. Long recordings handled via chunked sliding-window stitching.pyannote/brouhaharuns in an isolated subprocess venv (it needs the GitHub-onlybrouhaha-vad+ old pyannote); loads viaensure_hf_model.ASR / transcription overhaul
--aligner mms.ScriptLine.scorewhen a backend exposes it (verified: none of the current three do; plumbing is ready).Infra fixes (surfaced during validation)
numba>=0.60in all NeMo/Qwen subprocess venvs (qwen-asr, canary-qwen, nemo, nemo-diarization) — fixes thelibrosa→numba→llvmlite 0.36py3.12 backtrack.huggingface_hub<0.24in the Brouhaha venv;peftin the canary venv;ctranslate2on the CrisperWhisper transformers path.librosareverted from an explicit dep (bandwidth reimplemented ontorch.stft).Validation against ground truth (
audio_48khz_mono_16bits)Status / scope
specs/20260722-175022-scene-quality-utterance/(spec/plan/research/data-model/contracts/tasks) committed. US1–US3 implemented; US4 (utterance token-entropy) is partial (native-confidence plumbing only, as no backend exposes it); US5 (synthetic calibration harness) deferred.Caveats for reviewers
HF_TOKEN.🤖 Generated with Claude Code
Version
Published prerelease version:
1.3.1-alpha.35Changelog
🐛 Bug Fix
Authors: 3