hf-cache: stop HF Hub 429s under parallel batch (resolve-once + SHA-pin) - #539
Merged
Conversation
…ion checks The 429s come from from_pretrained/pipeline issuing a Hub HEAD/revision check on every call even for a cached, current model. Fix (MVP core): - _get_cached_commit_hash now resolves the real immutable commit SHA (refs/<ref> pointer or snapshot dir), not the mutable branch name; a full-SHA revision is passed through. - resolve_model(repo_id, revision) -> (sha, snapshot_path): download-once via ensure_hf_model, returns the SHA + local snapshot dir. - load_hf_resilient(loader, ..., repo_id, revision) resolves then calls the loader with revision=<sha>, local_files_only=True injected (pass_revision), so huggingface_hub's commit-hash shortcut returns cached files with zero network; pass_revision=False for loaders that take a local path instead. Tests-first (all pass): real-SHA resolution, SHA passthrough, resolve_model returns sha+path, load wrapper pins sha+local_files_only, and zero HfApi.model_info calls when cached. Backend migrations follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transformers ASR pipeline (where Qwen/Whisper "missing" 429s originated) now loads via load_hf_resilient: the mutable revision is dropped from the pipeline() kwargs, and the resolved commit SHA + local_files_only=True are injected, so a cached model triggers no per-call Hub version check. Behavior test (tests-first): asserts _get_hf_asr_pipeline routes through load_hf_resilient with repo_id/revision and never passes a mutable revision to pipeline() directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…el loads Subprocess-venv ASR workers (Qwen, Canary-Qwen, NeMo, Granite) load HF models in a fresh child process, so — unlike in-process — flipping HF_HUB_OFFLINE actually works there. hf_subprocess_env(repo_id, revision, also=..., base_env=...) stages each referenced model once (download-once via resolve_model) and returns the child env with HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE=1 so the worker's from_pretrained does no per-call Hub version check (the 429 source). Falls back to an unchanged env when a model can't be staged, so cold starts still download online. Tests-first: offline-when-all-cached, unchanged-when-uncacheable, and companion (`also`) staging (e.g. the Qwen forced aligner). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fix its 429) Qwen/Qwen3-ASR-1.7B (+ its Qwen3-ForcedAligner companion) is the backend that 429'd under parallel analyze_audio batches: the worker loaded both HF models with a plain _clean_subprocess_env(), so every job HEADed the Hub. Now the parent builds the child env via hf_subprocess_env(model, also=[(aligner, "main")]): both models are staged once and the child runs offline, so from_pretrained loads from cache with no Hub version check. Behavior test (tests-first): transcribe_with_qwen builds the worker env via hf_subprocess_env staging the ASR model + the forced-aligner companion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o_exists) check_hf_repo_exists' model branch caught every exception and returned False, so a transient 429/network error during verification made a real model report as "not found" (the confusing "Qwen missing" symptom). Now it matches the function's own non-model branch: only Repository/RevisionNotFoundError -> False (genuinely absent); GatedRepoError and transient errors are logged and re-raised so they surface as real, actionable errors instead of a false "missing". Tests-first: genuine-not-found -> False; transient -> logged + re-raised; gated -> re-raised. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nned loads The auto-align stage loads facebook/mms-1b-all (and any per-language Wav2Vec2 aligner) in-process for every timestamp-less ASR output, so each load HEADed the Hub. Route _load_mms_aligner and the per-language loader in align_transcriptions through load_hf_resilient: resolve ref->SHA once and load revision=<sha>, local_files_only=True, so cached aligners make no per-call Hub version check. Behavior test (tests-first): _load_mms_aligner loads processor + model via load_hf_resilient pinned to MMS_MODEL_ID. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eline) Regression found in the 400-subject stress run: load_hf_resilient injected local_files_only=True, but transformers.pipeline routes unknown kwargs into the model's generate params -> "ValueError: model_kwargs are not used by the model: ['local_files_only']" at inference, so Whisper failed on every lexical audio. A full-SHA revision alone triggers huggingface_hub's commit-hash shortcut (no HEAD), so local_files_only was never needed. Inject only revision=<sha>. Test updated to assert revision=<sha> is injected and local_files_only is NOT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(Groups A–C)
Extend the HF-429 fix beyond the ASR/alignment lane to the other model-load
sites, reusing the existing heartbeat-lock + version-resolution helpers
(resolve_model / load_hf_resilient / hf_subprocess_env). No new mechanism.
Group A (loaders that accept `revision` -> pin revision=<sha>):
- classification/huggingface.py (AST scene classification)
- speech_to_text/granite.py (Granite ASR)
- text_to_speech/huggingface.py (TTS pipeline)
- text/embeddings_extraction/{huggingface,sentence_transformers}.py
- ssl_embeddings/self_supervised_features.py (transformers feat-extractor + model)
- classification/speech_emotion_recognition/api.py (EmotionModel + feat-extractor)
Group B (SpeechBrain/pyannote -> stage under lock, load from immutable snapshot
path / pin pyannote revision; also fixes latent `revision=f"{None}"`):
- speaker_embeddings/speechbrain.py (ECAPA/ResNet)
- speech_enhancement/speechbrain.py (SepFormer)
- speaker_diarization/pyannote.py + voice_activity_detection/pyannote_vad.py
- ssl_embeddings + SER SpeechBrain loaders
Group C (subprocess -> hf_subprocess_env stages model + sets HF_HUB_OFFLINE):
- workflows/audio_analysis/pii_subprocess.py (GLiNER, only when gliner requested)
- SER continuous-emotion subprocess worker
Validated: SpeechBrain snapshot-path load (ECAPA -> 192-D) and transformers
revision=sha load (MiniLM) work end-to-end; ruff + 12/12 module imports + the
dependencies helper tests pass.
Residual (not migrated, tracked separately): SER metadata-introspection calls
(_load_config_cached AutoConfig/hf_hub_download; _peek_head_final_layer
HfApi.list_repo_files / get_safetensors_metadata) — online API peeks that don't
fit the model-load helper pattern; already memoized + per-(path,rev) locked and
not on the default analyze_audio path. #536-only files (frame_posteriors,
brouhaha, crisperwhisper) handled on the stress branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s_env Close the ASR-lane gap on the alpha branch: canary_qwen.py and nemo.py are alpha files but were only fixed on the stress branch, not here. Stage the model once (cross-process heartbeat lock) + run the worker offline so SALM/ASRModel from_pretrained make no per-call Hub version check — the 429 source under batch. Same proven pattern as qwen.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- speaker_diarization/nvidia.py: stage the Sortformer model + run the worker offline via hf_subprocess_env (was a raw subprocess load — surfaced by the new guard test below). - tests/utils/hf_load_coverage_test.py: static inventory guard. Walks src/senselab, finds every in-process HF loader (from_pretrained/from_hparams/ pipeline/snapshot_download/hf_hub_download/SentenceTransformer/Inference) and every subprocess worker that loads a model, and FAILS when one appears in a file not on the reviewed allowlist. Forces any newly-added HF functionality to be routed through resolve_model / load_hf_resilient / hf_subprocess_env and consciously allowlisted. Also asserts allowlisted files still reference a helper (catches silent regressions) and that the allowlists carry no stale entries. Verified it fails on a planted raw pipeline() load. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mits) - ssl_embeddings/self_supervised_features.py: only SHA-pin when cache_dir is None. resolve_model stages into the default HF cache, so a custom cache_dir would miss it and re-download (double download); the custom-cache_dir path now keeps prior ref-based behavior instead of staging into the wrong place. The common cache_dir=None path keeps the full fix. - tests/utils/hf_load_coverage_test.py: document the guard's two by-design limitations (file-granular, not per-load inside an already-reviewed file; subprocess detection assumes worker string + launch share a module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wilke0818
marked this pull request as ready for review
July 27, 2026 20:27
#536 (scene-quality/utterance) + #538 (cached-inference refactor) merged to alpha, so their model-load sites now live here too. Route them through the helpers (cherry-picked from the hf-cache-on-536 integration branch), guard-driven: In-process (SHA-pin / snapshot-path): - voice_activity_detection/frame_posteriors.py (use the SHA ensure_hf_model returns) - workflows/audio_analysis/adaptive/audio_io.py (SepFormer via snapshot path) - workflows/audio_analysis/adaptive/backends.py (ASR pipeline via load_hf_resilient) Subprocess (hf_subprocess_env): - scene_quality/brouhaha.py (gated; token forwarded, online fallback preserved) - speech_to_text/crisperwhisper.py (online fallback preserved) Guard-test allowlist updated to cover all five; PR #1 now covers every HF load site on alpha. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… patching)
Three unrelated CI failures on the merged branch:
1. load_hf_resilient's named `token` param shadowed the loader's token: the HF
ASR pipeline put `token` in pipeline_kwargs, it bound to the helper's param
(used only for resolve_model) and never reached `pipeline()`. Forward `token`
to the loader too (setdefault). Fixes test_hf_asr_pipeline_forwards_hf_token.
2. test_check_hf_repo_exists_false raised a generic Exception and expected False,
but check_hf_repo_exists now (correctly) re-raises transient errors and only
returns False for a definitive RepositoryNotFoundError — update the test to
raise that (built with a mock response, since it's an HfHubHTTPError).
3. dependencies_test.py monkeypatched string targets
("senselab.utils.dependencies.X"); under the full suite `senselab.utils` isn't
resolvable as an attribute mid-run, so pytest's resolver raised AttributeError.
Patch via the imported module object (`monkeypatch.setattr(dep, "X", ...)`),
which needs no attribute walk. Fixes all 7 dependencies_test failures.
Verified: the 8 formerly-failing tests + full utils dir + speech_to_text + guard
pass locally; ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Need
Under a parallel batch (e.g. a SLURM array loading models across hundreds of
processes at once), every HF model load issues a per-call Hub HEAD/revision
check, and the Hub rate-limits the burst (HTTP 429). This surfaced as ASR
backends intermittently "missing" in
analyze_audioruns.Validated at scale: with this fix the ASR/alignment lane ran ~16k audios across
a 400-subject batch with zero 429s (Qwen — the original offender — plus
Canary and Whisper all clean).
Plan
Resolve a model's ref to an immutable commit SHA once (download-once via the
existing cross-process heartbeat lock in
ensure_hf_model), then pin every loadto that SHA so cached files load with no Hub HEAD:
resolve_model(repo_id, revision) -> (sha, snapshot_path)load_hf_resilient(loader, ..., repo_id, revision)— in-process loaders thataccept
revision(injectsrevision=<sha>; nolocal_files_only, whichbreaks
transformers.pipeline).hf_subprocess_env(repo_id, revision, ...)— subprocess-venv backends: stageonce, run the worker with
HF_HUB_OFFLINE=1.This PR routes every HF model-load site on alpha through those helpers
(merges current alpha, which now includes #536 scene-quality/ASR + #538
cached-inference refactor):
ASR, TTS, text embeddings (HF + sentence-transformers), SSL, SER weights,
SpeechBrain speaker embeddings (ECAPA/ResNet) + SepFormer enhancement, pyannote
diarization + VAD, MMS forced aligner, Whisper pipeline, pyannote
segmentation frame-posteriors, the adaptive workflow's SepFormer + ASR loads.
hf_subprocess_env): Qwen, Canary, NeMo, Sortformerdiarization, continuous-SER, GLiNER PII, Brouhaha scene-quality, CrisperWhisper.
Plus a guard test (
tests/utils/hf_load_coverage_test.py) that staticallyinventories every HF load site and fails when a new one is added to a file
not on the reviewed allowlist — forcing future HF functionality to be routed
through the helpers and consciously reviewed. (It already caught otherwise-missed
gaps: Sortformer and the adaptive workflow's loads.)
Notes:
cache_dirkeeps prior behavior (resolve_model stages into the default cache).
named repo; sub-model loads are a separate follow-up.
🤖 Generated with Claude Code
Version
Published prerelease version:
1.3.1-alpha.38Changelog
🐛 Bug Fix
Authors: 3