fix(voiceprints)!: repair the silent rebuild and run it automatically - #151
Merged
Conversation
The voiceprint rebuild reported success having done nothing. A run over a library with two stale voiceprints finished in 0.14 seconds, too fast to have loaded the embedding model or opened any audio, and returned speakers_rebuilt: 0. Celery logged the task as succeeded, the API returned 200, and the Settings prompt stayed up, so running it again produced an identical no-op. Both stale speakers owned no attributable speech. One belonged to a recording whose utterances all carry a null recording_speaker_id, the other to a recording where five of six speakers had utterance rows and it did not. _speaker_time_ranges consulted the transcript segment blob only when no speaker in the recording had utterances, so the partially covered recording never reached the fallback, and the fully uncovered one reached it but found no segment carrying that speaker's label. Both then hit a bare continue with no counter and no log line. The fallback is now applied per speaker rather than per recording, and every path that cannot rebuild resolves the row instead of skipping it. A stale voiceprint that can never be re-extracted, whether from absent speech, missing audio, a missing recording, or an extraction that ran and produced nothing, is cleared. It could not be scored against anything, so nothing that was working is lost, and clearing is what lets a run converge. Transient failures are deliberately excluded: an exception leaves the voiceprint stale and is counted separately, so a decode hiccup cannot destroy something a later run could rebuild. The summary now reports cleared, retryable and missing-recording counts, and every cleared voiceprint is logged with its reason. Two further defects surfaced while tracing this. The stale query was not scoped to the requesting user, so the reported count described every user's library and the per-run recording limit was sliced off the global set before the user filter ran. A busy neighbour could consume the whole budget while the requesting user's recordings went untouched and the summary still reported nothing remaining. Scoping moved into the SQL. Separately, both embedding columns used Column(JSONB), which serialises a cleared value to JSON null rather than SQL NULL, so the row still satisfied embedding IS NOT NULL while reading back as None in Python. The database and the application disagreed about whether a voiceprint existed, which is the origin of ten such rows in this project's own library and of a stale count that read 38 in SQL and 28 in the task. Both columns now set none_as_null. The rebuild is also no longer user-triggered. Staleness is a maintenance obligation Nojoin owes itself after changing its own extraction method, not a state a user can act on. The prompt described a condition whose only available response was pressing a button, with no way to confirm it had worked. A Beat entry now runs the sweep every six hours. The cost the operator trigger was protecting against, an unbounded embedding run on hardware the user owns, is handled by bounding the work rather than by asking permission. Each tick processes at most AUTOMATIC_VOICEPRINT_REBUILD_LIMIT recordings on the GPU lane, where it is already serialised behind live capture and final processing, so a large library converges over several ticks. The task queries first and returns in milliseconds when nothing is stale, which is the steady state, so scheduling it unconditionally costs nothing. Automation and the clearing policy are load-bearing for each other. On the previous behaviour a bounded repeating sweep would never terminate, because unrebuildable rows stayed stale and every tick would rediscover and re-attempt the same recordings indefinitely. BREAKING CHANGE: GET /speakers/voiceprints/method-status and POST /speakers/voiceprints/rebuild are removed, along with the Settings panel and frontend API surface that consumed them. Voiceprint rebuilding is now automatic scheduled maintenance with no request-triggered entry point. Refs: docs/adr/0005-versioned-voiceprint-extraction.md
15 tasks
Valtora
added a commit
that referenced
this pull request
Jul 28, 2026
Bump docs/VERSION to 2.0.0 ahead of tagging. The release workflow fails fast if the pushed tag does not match this value exactly, so the bump has to land on main before v2.0.0 is created. A major bump is correct for this range. The voiceprint fix removes GET /speakers/voiceprints/method-status and POST /speakers/voiceprints/rebuild outright under a BREAKING CHANGE footer, replacing the operator trigger with scheduled maintenance, and the settings work moves every category onto its own route. The upgrade is also not pull and recreate alone: operators must remove the backup_temp mounts and the second recordings bind from their Compose file, because a stale mount reinstates the shared, permanent /tmp that this range fixes. There are no new Alembic revisions in v1.7.0..HEAD, so the upgrade carries no schema migration. The full release validation set passes locally at this commit: 1142 backend tests, frontend lint, 320 frontend unit tests, the production build, and the docs, Alembic and held-pin validators. All four release images (api, worker, worker-io, frontend) were built from 22219f2 and scanned with the release gate flags, meaning CRITICAL and HIGH, fixed findings only, against .trivyignore. All four pass, so no new .trivyignore entries are required. Refs: #151, #154, docs/DEPLOYMENT.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The voiceprint rebuild reported success having done nothing, and there was no way to tell from the outside. A run over a library with two stale voiceprints finished in 0.14 seconds, far too fast to have loaded the embedding model or opened any audio, and returned
speakers_rebuilt: 0. Celery logged the task assucceeded, the API returned 200, and the Settings prompt stayed up, so a second run produced an identical no-op.Both stale speakers owned no attributable speech.
_speaker_time_rangesconsulted the transcript segment blob only when no speaker in the recording had utterance rows, so a recording with partial utterance coverage never reached the fallback at all, and a fully uncovered one reached it but found no segment carrying that speaker's label. Both then hit a barecontinuewith no counter and no log line, which is indistinguishable from having no work to do.Changes:
speakers_cleared_unrebuildable,speakers_failed_retryable,recordings_missing,people_cleared_unrebuildable, plus a log line per cleared voiceprint with its reason.Column(JSONB)->Column(JSONB(none_as_null=True))on both embedding columns. AssigningNonewas serialising to a JSONnullrather than SQL NULL, so a cleared row still satisfiedembedding IS NOT NULLwhile reading back asNonein Python. That is the origin of ten such rows in this project's own library and of a stale count that read 38 in SQL and 28 in the task.Staleness is a maintenance obligation Nojoin owes itself after changing its own extraction method, not a state a user can act on. The prompt described a condition whose only available response was pressing a button, with no way to confirm it had worked. The cost the operator trigger was protecting against, an unbounded embedding run on hardware the user owns, is handled by bounding the work instead: each tick processes at most
AUTOMATIC_VOICEPRINT_REBUILD_LIMIT(25) recordings on the GPU lane, where it is already serialised behind live capture and final processing, so a large library converges over several ticks. The task queries first and returns in milliseconds when nothing is stale, which is the steady state, so scheduling it unconditionally costs nothing.Automation and the clearing policy are load-bearing for each other. On the previous behaviour a bounded repeating sweep would never terminate, because unrebuildable rows stayed stale and every tick would rediscover and re-attempt the same recordings indefinitely.
No new dependencies.
Type of change
Breaking:
GET /speakers/voiceprints/method-statusandPOST /speakers/voiceprints/rebuildare removed. Nothing but the deleted Settings panel consumed them.Checks run
source .venv/bin/activate && pytest(1101 passed)python scripts/check.py(lint, format, whitespace, filesize, heldpins, typecheck, docs, alembic, tests -- all OK)cd frontend && npm run lintcd frontend && npm run test(293 passed, 51 files)cd frontend && npm run buildpython3 scripts/validate_docs.pypython3 scripts/validate_alembic.pyMigration impact
none_as_nullis a SQLAlchemy type argument, not a schema change. The column staysJSONB. Rows that already hold a JSONnullkeep it, which is why the Python truthiness guard remains the authority in the task and now carries a comment saying why it is not redundant with the SQL predicate.Documentation impact
docs/USAGE.mdnow states that rebuilding happens on its own with nothing to press, and names the trade-off: identification can stay degraded for a while after an upgrade while the library converges.docs/adr/0005-versioned-voiceprint-extraction.mdgets two amendments, since it explicitly decided both of the things this PR reverses (unrebuildable rows "stay stale", and the rebuild "is operator-triggered, not automatic on upgrade") and named both deleted endpoints.Security impact
Manual verification
Diagnosed against the live deployment before writing any code: read the worker-gpu container logs for the real task run, then queried the production database to confirm the two stale speakers had no utterance rows and no matching transcript segment label, that
snippet_start/snippet_endwere NULL so no alternative range source existed, and that ten rows held a JSONnullembedding.The 12 new tests in
backend/tests/test_voiceprint_rebuild.pywere verified to fail against the old implementation by stashing the fix and re-running: 11 of 12 failed, including a convergence test asserting a second run has nothing left to do, which was the reported symptom. The twelfth (utterance ranges winning over transcript segments) passes both ways by design, since that precedence was already correct.Not manually verified: the Beat entry firing on a real six-hour cadence in a running deployment. Its wiring and per-run bound are covered by tests in
backend/tests/test_celery_beat_schedule.py. No capture, recording context-menu, or auth paths are touched.