Skip to content

Issue 2: Model should be volume-mounted, not downloaded inside the container #2

Description

@arpan-52

Body:
Problem

Currently, if MODEL_PATH is not set, start.sh warns and proceeds — the container
then downloads the GGUF from HuggingFace on every cold start. A Q4_K_XL GGUF for
Qwen3.5-4B is ~3 GB. This is slow, uses bandwidth, and the download is lost when
the container is removed.

Expected behaviour

The model GGUF should live on the host and be passed in via MODEL_PATH:

MODEL_PATH=/home/user/models/qwen3.5-4b-q4_k_xl.gguf \
MS_PATH=/data/3C129_1.ms \
./start.sh

start.sh mounts it read-only at /models/model.gguf inside the container.

Proposed change

  • Make MODEL_PATH required in start.sh (fail with a clear error if unset).
  • Document the one-time model download step in README:

one-time: download the model to your host

huggingface-cli download unsloth/Qwen3.5-4B-GGUF \
Qwen3.5-4B-UD-Q4_K_XL.gguf \
--local-dir ~/models/

  • Remove HuggingFace download logic from the entrypoint/container entirely.

Why

  • Container storage is ephemeral — downloaded models are lost on ./stop.sh.
  • Re-downloading 3 GB on every fresh start is unacceptable for dev iteration.
  • Volume-mounting keeps the model on the host where it persists across rebuilds and restarts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions