Skip to content

fix(voiceprints)!: repair the silent rebuild and run it automatically - #151

Merged
Valtora merged 1 commit into
mainfrom
fix/voiceprint-rebuild-silent-noop
Jul 27, 2026
Merged

fix(voiceprints)!: repair the silent rebuild and run it automatically#151
Valtora merged 1 commit into
mainfrom
fix/voiceprint-rebuild-silent-noop

Conversation

@Valtora

@Valtora Valtora commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Description

The voiceprint rebuild reported success having done nothing, and there was no way to tell from the outside. A run over a library with two stale voiceprints finished in 0.14 seconds, far too fast to have loaded the embedding model or opened any audio, and returned speakers_rebuilt: 0. Celery logged the task as succeeded, the API returned 200, and the Settings prompt stayed up, so a second run produced an identical no-op.

Both stale speakers owned no attributable speech. _speaker_time_ranges consulted the transcript segment blob only when no speaker in the recording had utterance rows, so a recording with partial utterance coverage never reached the fallback at all, and a fully uncovered one reached it but found no segment carrying that speaker's label. Both then hit a bare continue with no counter and no log line, which is indistinguishable from having no work to do.

Changes:

  • Per-speaker segment fallback. Applied per speaker rather than per recording, so a recording processed across a pipeline change no longer strands its partially covered speakers.
  • Unrebuildable voiceprints are cleared. No attributable speech, missing audio, missing recording, or an extraction that ran and produced nothing. Such a vector could never be scored against anything, so nothing that was working is lost, and clearing is what lets a run converge. The speaker or person row survives; only the unusable vector goes.
  • Transient failures are excluded from that rule. An exception leaves the voiceprint stale and is counted separately, so a decode hiccup cannot destroy something a later run could rebuild.
  • Every path is counted and logged. speakers_cleared_unrebuildable, speakers_failed_retryable, recordings_missing, people_cleared_unrebuildable, plus a log line per cleared voiceprint with its reason.
  • User-scoped stale query. Scoping moved into the SQL. Previously the reported count described every user's library, and the per-run recording limit was sliced off the global set before the user filter ran, so a busy neighbour could consume the whole budget while the requesting user's recordings went untouched and the summary still reported nothing remaining.
  • Column(JSONB) -> Column(JSONB(none_as_null=True)) on both embedding columns. Assigning None was serialising to a JSON null rather than SQL NULL, so a cleared row still satisfied embedding IS NOT NULL while reading back as None in Python. That is the origin of ten such rows in this project's own library and of a stale count that read 38 in SQL and 28 in the task.
  • The rebuild is now automatic scheduled maintenance. A Beat entry runs it every six hours; the Settings panel and both endpoints are removed.

Staleness is a maintenance obligation Nojoin owes itself after changing its own extraction method, not a state a user can act on. The prompt described a condition whose only available response was pressing a button, with no way to confirm it had worked. The cost the operator trigger was protecting against, an unbounded embedding run on hardware the user owns, is handled by bounding the work instead: each tick processes at most AUTOMATIC_VOICEPRINT_REBUILD_LIMIT (25) recordings on the GPU lane, where it is already serialised behind live capture and final processing, so a large library converges over several ticks. The task queries first and returns in milliseconds when nothing is stale, which is the steady state, so scheduling it unconditionally costs nothing.

Automation and the clearing policy are load-bearing for each other. On the previous behaviour a bounded repeating sweep would never terminate, because unrebuildable rows stayed stale and every tick would rediscover and re-attempt the same recordings indefinitely.

No new dependencies.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behaviour)
  • Documentation update

Breaking: GET /speakers/voiceprints/method-status and POST /speakers/voiceprints/rebuild are removed. Nothing but the deleted Settings panel consumed them.

Checks run

  • Backend tests: source .venv/bin/activate && pytest (1101 passed)
  • Python quality: python scripts/check.py (lint, format, whitespace, filesize, heldpins, typecheck, docs, alembic, tests -- all OK)
  • Frontend lint: cd frontend && npm run lint
  • Frontend unit tests: cd frontend && npm run test (293 passed, 51 files)
  • Frontend build: cd frontend && npm run build
  • Docs validation: python3 scripts/validate_docs.py
  • Alembic validation: python3 scripts/validate_alembic.py

Migration impact

  • No database migration in this PR.
  • Adds an Alembic migration.

none_as_null is a SQLAlchemy type argument, not a schema change. The column stays JSONB. Rows that already hold a JSON null keep it, which is why the Python truthiness guard remains the authority in the task and now carries a comment saying why it is not redundant with the SQL predicate.

Documentation impact

  • No documentation change required.
  • Updated the relevant guide(s) in the same PR.

docs/USAGE.md now states that rebuilding happens on its own with nothing to press, and names the trade-off: identification can stay degraded for a while after an upgrade while the library converges. docs/adr/0005-versioned-voiceprint-extraction.md gets two amendments, since it explicitly decided both of the things this PR reverses (unrebuildable rows "stay stale", and the rebuild "is operator-triggered, not automatic on upgrade") and named both deleted endpoints.

Security impact

  • No security-sensitive change.
  • Touches auth, tokens, encryption, capture ownership, or exposure.

Manual verification

Diagnosed against the live deployment before writing any code: read the worker-gpu container logs for the real task run, then queried the production database to confirm the two stale speakers had no utterance rows and no matching transcript segment label, that snippet_start/snippet_end were NULL so no alternative range source existed, and that ten rows held a JSON null embedding.

The 12 new tests in backend/tests/test_voiceprint_rebuild.py were verified to fail against the old implementation by stashing the fix and re-running: 11 of 12 failed, including a convergence test asserting a second run has nothing left to do, which was the reported symptom. The twelfth (utterance ranges winning over transcript segments) passes both ways by design, since that precedence was already correct.

Not manually verified: the Beat entry firing on a real six-hour cadence in a running deployment. Its wiring and per-run bound are covered by tests in backend/tests/test_celery_beat_schedule.py. No capture, recording context-menu, or auth paths are touched.

The voiceprint rebuild reported success having done nothing. A run over
a library with two stale voiceprints finished in 0.14 seconds, too fast
to have loaded the embedding model or opened any audio, and returned
speakers_rebuilt: 0. Celery logged the task as succeeded, the API
returned 200, and the Settings prompt stayed up, so running it again
produced an identical no-op.

Both stale speakers owned no attributable speech. One belonged to a
recording whose utterances all carry a null recording_speaker_id, the
other to a recording where five of six speakers had utterance rows and
it did not. _speaker_time_ranges consulted the transcript segment blob
only when no speaker in the recording had utterances, so the partially
covered recording never reached the fallback, and the fully uncovered
one reached it but found no segment carrying that speaker's label. Both
then hit a bare continue with no counter and no log line.

The fallback is now applied per speaker rather than per recording, and
every path that cannot rebuild resolves the row instead of skipping it.
A stale voiceprint that can never be re-extracted, whether from absent
speech, missing audio, a missing recording, or an extraction that ran
and produced nothing, is cleared. It could not be scored against
anything, so nothing that was working is lost, and clearing is what
lets a run converge. Transient failures are deliberately excluded: an
exception leaves the voiceprint stale and is counted separately, so a
decode hiccup cannot destroy something a later run could rebuild. The
summary now reports cleared, retryable and missing-recording counts,
and every cleared voiceprint is logged with its reason.

Two further defects surfaced while tracing this. The stale query was
not scoped to the requesting user, so the reported count described
every user's library and the per-run recording limit was sliced off the
global set before the user filter ran. A busy neighbour could consume
the whole budget while the requesting user's recordings went untouched
and the summary still reported nothing remaining. Scoping moved into
the SQL. Separately, both embedding columns used Column(JSONB), which
serialises a cleared value to JSON null rather than SQL NULL, so the
row still satisfied embedding IS NOT NULL while reading back as None in
Python. The database and the application disagreed about whether a
voiceprint existed, which is the origin of ten such rows in this
project's own library and of a stale count that read 38 in SQL and 28
in the task. Both columns now set none_as_null.

The rebuild is also no longer user-triggered. Staleness is a
maintenance obligation Nojoin owes itself after changing its own
extraction method, not a state a user can act on. The prompt described
a condition whose only available response was pressing a button, with
no way to confirm it had worked. A Beat entry now runs the sweep every
six hours. The cost the operator trigger was protecting against, an
unbounded embedding run on hardware the user owns, is handled by
bounding the work rather than by asking permission. Each tick processes
at most AUTOMATIC_VOICEPRINT_REBUILD_LIMIT recordings on the GPU lane,
where it is already serialised behind live capture and final
processing, so a large library converges over several ticks. The task
queries first and returns in milliseconds when nothing is stale, which
is the steady state, so scheduling it unconditionally costs nothing.

Automation and the clearing policy are load-bearing for each other. On
the previous behaviour a bounded repeating sweep would never terminate,
because unrebuildable rows stayed stale and every tick would rediscover
and re-attempt the same recordings indefinitely.

BREAKING CHANGE: GET /speakers/voiceprints/method-status and POST
/speakers/voiceprints/rebuild are removed, along with the Settings
panel and frontend API surface that consumed them. Voiceprint
rebuilding is now automatic scheduled maintenance with no
request-triggered entry point.

Refs: docs/adr/0005-versioned-voiceprint-extraction.md
@Valtora Valtora closed this Jul 27, 2026
@Valtora Valtora reopened this Jul 27, 2026
@Valtora
Valtora merged commit 5ca9b22 into main Jul 27, 2026
33 of 34 checks passed
@Valtora
Valtora deleted the fix/voiceprint-rebuild-silent-noop branch July 27, 2026 22:56
@Valtora Valtora mentioned this pull request Jul 28, 2026
15 tasks
Valtora added a commit that referenced this pull request Jul 28, 2026
Bump docs/VERSION to 2.0.0 ahead of tagging. The release workflow fails
fast if the pushed tag does not match this value exactly, so the bump has
to land on main before v2.0.0 is created.

A major bump is correct for this range. The voiceprint fix removes
GET /speakers/voiceprints/method-status and POST
/speakers/voiceprints/rebuild outright under a BREAKING CHANGE footer,
replacing the operator trigger with scheduled maintenance, and the
settings work moves every category onto its own route. The upgrade is
also not pull and recreate alone: operators must remove the backup_temp
mounts and the second recordings bind from their Compose file, because a
stale mount reinstates the shared, permanent /tmp that this range fixes.

There are no new Alembic revisions in v1.7.0..HEAD, so the upgrade
carries no schema migration.

The full release validation set passes locally at this commit: 1142
backend tests, frontend lint, 320 frontend unit tests, the production
build, and the docs, Alembic and held-pin validators. All four release
images (api, worker, worker-io, frontend) were built from 22219f2 and
scanned with the release gate flags, meaning CRITICAL and HIGH, fixed
findings only, against .trivyignore. All four pass, so no new
.trivyignore entries are required.

Refs: #151, #154, docs/DEPLOYMENT.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant