feat(settings): ask before downloading a newly selected model (#152)

* feat(settings): ask before downloading a newly selected model

Saving a transcription model change queued a download as a side effect.
That download runs on the GPU lane, the same lane that serves live
transcription, so on a single-card host it could sit in front of a
meeting; nothing in the UI said it had started, and the setup wizard's
promise to "check progress later in Settings" was never honoured.

Preparation is now explicit. Selecting a model that is not on the server
prompts for a choice: fetch it now so it is ready for the next
recording, or leave it to the lazy fetch on first use, which delays live
transcription and Meeting Edge until the download completes. Selecting a
model already on disk saves silently, so the prompt only appears when
there is something to download.

A new admin-only POST /system/models/prepare carries the request, with
targets for the active selection, the core batch, and each ONNX ASR
model, so a single missing row in Model dependencies can be repaired on
its own. It refuses a second request with 409 while one is in flight,
and resolves the model from the admin's own settings rather than the
install config, since the transcription keys are user-scoped and reading
config alone would prepare the install default instead.

Model dependencies gains a Download action on every missing row plus a
live progress strip, so declining the prompt is recoverable and a
running preparation is visible however it was started.

Saving settings through the API no longer prepares models for any
caller. Lazy fetch on first use still covers that path.

Refs: docs/DEPLOYMENT.md, docs/ADMIN.md, docs/USAGE.md

* fix(models): detect cached ONNX ASR models by their repo name

Canary was reported as missing however many times it was downloaded.
The status check matched the Nojoin model id against Hugging Face cache
directory names, but onnx-asr caches nemo-canary-1b-v2 under the repo
istupakov/canary-1b-v2-onnx, so the id never appeared in the directory
name. Parakeet escaped the bug only because its id happens to be a
substring of its repo name.

Match a fragment of the repo name instead, held in one named map with
the divergence written down so it is not "corrected" back to the model
id later. Resolving the id through onnx-asr would be exact, but that
would pull the ASR stack into the API process, which the torch boundary
test exists to prevent.

This also unblocks deletion and the admin health card, both of which
read the same status: a model reported as missing has no resolvable path
to delete, and the transcription component showed Canary as uncached.

Refs: backend/preload_models.py

* fix(models): delete cached models from a worker, not the API

Deleting a model failed on every Docker install with EROFS. The API
mounts the shared model volume read-only, by design, so the delete could
never have worked from there: the failure was structural rather than a
bad path.

Dispatch the delete to the io lane, which mounts that volume read-write
along with the other worker lanes. The task resolves the path itself
rather than accepting one from the API, because the same volume is
mounted at a different path in each container, and a delete sink that
trusts a caller-supplied path is worth not building. It returns a status
dict instead of raising, since the JSON serialiser would not carry the
difference between "not found" and "refused" across the boundary.

The request still waits for the result, so the UI can keep refreshing
model status the moment it returns, with a 60 second ceiling that fails
the request rather than holding a thread when no worker answers.

Deleting a model now needs a running worker; that is documented.

Refs: docs/DEPLOYMENT.md, docs/ADMIN.md

Author: Valtora

backend/api/v1/endpoints/settings.py

@@ -11,7 +11,6 @@
 from backend.models.notes_template import NotesTemplate, NotesTemplateScope
 from backend.models.recording import Recording, RecordingStatus
 from backend.models.user import User
-from backend.services.model_preparation import enqueue_model_preparation
 from backend.utils.config_manager import (
     APP_THEMES,
     INSTALL_WIDE_AI_SETTING_KEYS,
@@ -517,31 +516,10 @@ async def _save_user_settings(
     if meeting_edge_keys.intersection(update_data):
         await _dispatch_meeting_edge_refresh_for_active_recordings(db, current_user)
 
-    model_keys = {
-        "whisper_model_size",
-        "transcription_backend",
-        "parakeet_model",
-        "canary_model",
-    }
-    if is_admin and model_keys.intersection(update_data):
-        prepared_settings = dict(current_settings)
-        try:
-            enqueue_model_preparation(
-                whisper_model_size=prepared_settings.get("whisper_model_size"),
-                transcription_backend=prepared_settings.get("transcription_backend"),
-                parakeet_model=prepared_settings.get("parakeet_model"),
-                canary_model=prepared_settings.get("canary_model"),
-                include_core=(
-                    "whisper_model_size" in update_data
-                    or update_data.get("transcription_backend") == "whisper"
-                ),
-            )
-        except Exception as e:  # noqa: BLE001
-            logger.error(
-                "Failed to queue model preparation after settings update: %s",
-                e,
-                exc_info=True,
-            )
+    # Changing the transcription model deliberately does not download anything.
+    # Preparation runs on the GPU lane, so it is requested explicitly through
+    # POST /system/models/prepare after the admin has been asked; a model that is
+    # never prepared is still fetched lazily on first use.
 
     return await _merge_settings(current_user.settings, db)
 

backend/api/v1/endpoints/system.py

@@ -2,6 +2,7 @@
 import logging
 from typing import Any, Optional
 
+from celery.exceptions import TimeoutError as CeleryTimeoutError
 from celery.result import AsyncResult
 from docker.client import DockerClient
 from docker.errors import DockerException, NotFound
@@ -35,6 +36,7 @@
     is_system_initialized,
     require_first_run_password,
 )
+from backend.celery_app import celery_app
 from backend.core.security import (
     MIN_PASSWORD_LENGTH,
     hash_user_password,
@@ -54,6 +56,11 @@
 router = APIRouter()
 logger = logging.getLogger(__name__)
 
+MODEL_DELETION_TASK = "backend.worker.tasks.delete_model_task"
+# Deleting a cached model is an rmtree on a local volume, so seconds; the ceiling
+# is here so an absent worker fails the request rather than holding the thread.
+MODEL_DELETION_TIMEOUT_S = 60
+
 RETIRED_COMPANION_RESPONSE = {
     "error": "companion_retired",
     "message": "The Nojoin Companion app has been retired. Please update your installation and use the web app for recording.",
@@ -581,6 +588,82 @@ async def get_models_status(
     return status
 
 
+class ModelPreparationRequest(BaseModel):
+    """Which assets to prepare.
+
+    ``active`` covers whatever the saved transcription settings actually need,
+    which is what the Settings prompt asks about. The remaining targets exist so
+    a single missing row in Model dependencies can be repaired on its own.
+    """
+
+    target: str = Field(default="active")
+
+    @field_validator("target")
+    @classmethod
+    def validate_target(cls, value: str) -> str:
+        allowed = {"active", "core", "parakeet", "canary"}
+        if value not in allowed:
+            raise ValueError(f"target must be one of {sorted(allowed)}")
+        return value
+
+
+@router.post("/models/prepare")
+async def prepare_models_endpoint(
+    payload: ModelPreparationRequest | None = None,
+    current_user: User = Depends(get_current_admin_user),
+) -> Any:
+    """
+    Queue preparation of local model assets.
+
+    Preparation is explicit: saving a transcription model change no longer
+    downloads anything by itself, because the preparation task runs on the GPU
+    lane and would otherwise queue in front of live work unannounced.
+    """
+    target = (payload or ModelPreparationRequest()).target
+
+    if is_download_in_progress():
+        raise HTTPException(
+            status_code=409,
+            detail="Model preparation is already running. Wait for it to finish.",
+        )
+
+    # The transcription keys are user-scoped rather than install-wide, so the
+    # admin's own row wins over config.json. Reading config alone would prepare
+    # whatever the install default happens to be, not the model just chosen.
+    user_settings = current_user.settings or {}
+
+    def effective(key: str, default: str) -> str:
+        value = user_settings.get(key)
+        return str(value) if value else str(config_manager.get(key, default))
+
+    if target == "active":
+        transcription_backend = effective("transcription_backend", "whisper")
+        include_core = transcription_backend == "whisper"
+    elif target == "core":
+        transcription_backend = "whisper"
+        include_core = True
+    else:
+        transcription_backend = target
+        include_core = False
+
+    try:
+        task_id = enqueue_model_preparation(
+            whisper_model_size=effective("whisper_model_size", "turbo"),
+            transcription_backend=transcription_backend,
+            parakeet_model=effective("parakeet_model", "parakeet-tdt-0.6b-v3"),
+            canary_model=effective("canary_model", "nemo-canary-1b-v2"),
+            include_core=include_core,
+        )
+    except Exception as e:  # noqa: BLE001
+        logger.error("Failed to queue model preparation: %s", e, exc_info=True)
+        raise HTTPException(
+            status_code=503,
+            detail="Could not queue model preparation. Is the worker running?",
+        )
+
+    return {"task_id": task_id, "target": target, "status": "queued"}
+
+
 @router.delete("/models/{model_name}")
 async def delete_model_endpoint(
     model_name: str,
@@ -589,25 +672,50 @@ async def delete_model_endpoint(
 ) -> Any:
     """
     Delete a specific model from the cache.
+
+    Dispatched to a worker: the API mounts the shared model volume read-only, so
+    deleting from this process fails with EROFS. The call still waits for the
+    result, because the UI refreshes model status as soon as it returns.
     """
-    from backend.preload_models import delete_model
+    del current_user
 
     if model_name not in ["whisper", "pyannote", "embedding", "parakeet", "canary"]:
         raise HTTPException(status_code=400, detail="Invalid model name")
 
     try:
-        success = delete_model(model_name, whisper_model_size=variant)
-        if success:
-            return {"message": f"Model {model_name} deleted successfully"}
-        else:
-            raise HTTPException(
-                status_code=404,
-                detail=f"Model {model_name} not found or could not be deleted",
-            )
-    except ValueError as e:
-        raise HTTPException(status_code=400, detail=str(e))
+        task = celery_app.send_task(
+            MODEL_DELETION_TASK,
+            kwargs={"model_name": model_name, "variant": variant},
+        )
+        result = await asyncio.to_thread(task.get, timeout=MODEL_DELETION_TIMEOUT_S)
+    except CeleryTimeoutError:
+        raise HTTPException(
+            status_code=504,
+            detail=(
+                "Model deletion did not finish in time. It may still be running "
+                "on the worker; refresh model status in a moment."
+            ),
+        )
     except Exception as e:  # noqa: BLE001
-        raise HTTPException(status_code=500, detail=str(e))
+        logger.error("Model deletion task failed for %s: %s", model_name, e)
+        raise HTTPException(
+            status_code=503,
+            detail="Could not reach the worker to delete the model.",
+        )
+
+    status = (result or {}).get("status")
+    message = (result or {}).get("message") or f"Model {model_name} deleted."
+
+    if status == "deleted":
+        return {"message": message}
+    if status == "not_found":
+        raise HTTPException(
+            status_code=404,
+            detail=f"Model {model_name} not found or could not be deleted",
+        )
+    if status == "forbidden":
+        raise HTTPException(status_code=400, detail=message)
+    raise HTTPException(status_code=500, detail=message)
 
 
 @router.post("/seed-demo")

backend/celery_app.py

@@ -200,6 +200,9 @@ def release_model_caches_after_task(**kwargs):
     "backend.worker.tasks.ensure_calendar_push_channels_task": {"queue": IO_QUEUE},
     "backend.worker.tasks.renew_calendar_push_channels_task": {"queue": IO_QUEUE},
     "backend.worker.tasks.cleanup_temp_recordings": {"queue": IO_QUEUE},
+    # Local disk work on the model volume. Belongs on a worker at all because
+    # the API mounts that volume read-only.
+    "backend.worker.tasks.delete_model_task": {"queue": IO_QUEUE},
     "backend.worker.tasks.send_telemetry_ping_task": {"queue": IO_QUEUE},
 }
 

backend/preload_models.py

@@ -40,6 +40,19 @@
     "turbo": "large-v3-turbo.pt",
 }
 
+# Hugging Face cache directory fragments for the ONNX ASR models, used to detect
+# them without importing onnx-asr into the API process.
+#
+# These are fragments of the *repo* name, not of the Nojoin model id, because the
+# two diverge: `nemo-canary-1b-v2` is cached as `models--istupakov--canary-1b-v2-onnx`,
+# with no `nemo-` prefix. Matching the Nojoin id reported Canary as permanently
+# missing however many times it was downloaded, which also made it undeletable,
+# since deletion resolves its path through the same status check.
+ONNX_ASR_CACHE_FRAGMENTS = {
+    "parakeet": "parakeet-tdt-0.6b-v3",
+    "canary": "canary-1b-v2",
+}
+
 
 def _is_onnx_asr_model_cached(model_substring: str) -> bool:
     """Check if an onnx-asr model is present in the Hugging Face hub cache."""
@@ -510,7 +523,7 @@ def check_model_status(whisper_model_size=None):
                     status["whisper"]["downloaded"] = True
                     status["whisper"]["path"] = default_filepath
 
-    # Check Parakeet
+    # Check the ONNX ASR models.
     # Best-effort detection: onnx-asr caches the model under the Hugging Face hub
     # cache. Detection is a directory-name match; the exact repo dir name may vary
     # by onnx-asr version, so this is treated as a heuristic, not authoritative.
@@ -525,35 +538,20 @@ def check_model_status(whisper_model_size=None):
     if default_hf_cache not in parakeet_hf_caches:
         parakeet_hf_caches.append(default_hf_cache)
 
-    for cache_dir in parakeet_hf_caches:
-        status["parakeet"]["checked_paths"].append(cache_dir)
-        if os.path.isdir(cache_dir):
-            try:
-                for entry in os.listdir(cache_dir):
-                    if "parakeet-tdt-0.6b-v3" in entry:
-                        status["parakeet"]["downloaded"] = True
-                        status["parakeet"]["path"] = os.path.join(cache_dir, entry)
-                        break
-            except OSError:
-                pass
-        if status["parakeet"]["downloaded"]:
-            break
-
-    # Check Canary
-    # Same best-effort HF-cache directory-name match as Parakeet above.
-    for cache_dir in parakeet_hf_caches:
-        status["canary"]["checked_paths"].append(cache_dir)
-        if os.path.isdir(cache_dir):
-            try:
-                for entry in os.listdir(cache_dir):
-                    if "nemo-canary-1b-v2" in entry:
-                        status["canary"]["downloaded"] = True
-                        status["canary"]["path"] = os.path.join(cache_dir, entry)
-                        break
-            except OSError:
-                pass
-        if status["canary"]["downloaded"]:
-            break
+    for status_key, fragment in ONNX_ASR_CACHE_FRAGMENTS.items():
+        for cache_dir in parakeet_hf_caches:
+            status[status_key]["checked_paths"].append(cache_dir)
+            if os.path.isdir(cache_dir):
+                try:
+                    for entry in os.listdir(cache_dir):
+                        if fragment in entry:
+                            status[status_key]["downloaded"] = True
+                            status[status_key]["path"] = os.path.join(cache_dir, entry)
+                            break
+                except OSError:
+                    pass
+            if status[status_key]["downloaded"]:
+                break
 
     for status_key, model_id in (
         ("pyannote", "pyannote/speaker-diarization-community-1"),

backend/tests/test_model_cache_detection.py

@@ -0,0 +1,46 @@
+"""Detecting cached ONNX ASR models on disk.
+
+The status heuristic matches Hugging Face cache directory names. It has to match
+the *repo* name rather than the Nojoin model id, because the two diverge:
+onnx-asr caches `nemo-canary-1b-v2` as `models--istupakov--canary-1b-v2-onnx`.
+Matching the Nojoin id reported Canary as missing however many times it was
+prepared, and made it undeletable with it, since deletion resolves its path
+through this same check.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from backend.preload_models import check_model_status
+
+# The directory names a real install ends up with, taken from a live cache.
+CACHED_REPOS = (
+    "models--istupakov--canary-1b-v2-onnx",
+    "models--istupakov--parakeet-tdt-0.6b-v3-onnx",
+)
+
+
+@pytest.mark.parametrize("model", ["parakeet", "canary"])
+def test_a_downloaded_onnx_model_is_reported_as_present(model, monkeypatch, tmp_path):
+    hub = tmp_path / "hub"
+    hub.mkdir()
+    for repo in CACHED_REPOS:
+        (hub / repo).mkdir()
+    monkeypatch.setenv("HF_HOME", str(tmp_path))
+
+    status = check_model_status(whisper_model_size="turbo")
+
+    assert status[model]["downloaded"] is True
+    assert status[model]["path"].startswith(str(hub))
+
+
+@pytest.mark.parametrize("model", ["parakeet", "canary"])
+def test_an_empty_cache_reports_the_model_as_missing(model, monkeypatch, tmp_path):
+    (tmp_path / "hub").mkdir()
+    monkeypatch.setenv("HF_HOME", str(tmp_path))
+
+    status = check_model_status(whisper_model_size="turbo")
+
+    assert status[model]["downloaded"] is False
+    assert status[model]["path"] is None