Machine Learning Image docker image won't start

[Keksmo]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Every recent machine learning image (v1.141.1 is the newest one that works as intended) throws this error when starting the container. I've tried repulling the container, image and volume, nothing changed the issue.

The OS that Immich Server is running on

Debian

Version of Immich Server

v2.0.0

Version of Immich Mobile App

v1.143.1

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:v1.141.1
#    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - ./model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: redis
    image: docker.io/redis:6.2-alpine@sha256:148bb5411c184abd288d9aaed139c98123eeb8824c5d3fce03cf721db58066d8
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:739cdd626151ff1f796dc95a6591b55a714f341c737e27f045019ceabf8e8c52
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=<redacted>

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Pull the machine_learning image
2. Start a container with the image

Relevant log output

immich_machine_learning  | Traceback (most recent call last):
immich_machine_learning  |   File "<frozen runpy>", line 198, in _run_module_as_main
immich_machine_learning  |   File "<frozen runpy>", line 88, in _run_code
immich_machine_learning  |   File "/usr/src/immich_ml/__main__.py", line 6, in <module>
immich_machine_learning  |     from .config import log, non_prefixed_settings, settings
immich_machine_learning  |   File "/usr/src/immich_ml/config.py", line 8, in <module>
immich_machine_learning  |     from gunicorn.arbiter import Arbiter
immich_machine_learning  |   File "/opt/venv/lib/python3.11/site-packages/gunicorn/arbiter.py", line 15, in <module>
immich_machine_learning  |     from gunicorn import sock, systemd, util
immich_machine_learning  |   File "/opt/venv/lib/python3.11/site-packages/gunicorn/sock.py", line 13, in <module>
immich_machine_learning  |     from gunicorn import util
immich_machine_learning  |   File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 4, in <module>
immich_machine_learning  |     import ast
immich_machine_learning  | SyntaxError: source code string cannot contain null bytes
immich_machine_learning exited with code 1 (restarting)

Additional information

No response

[HeathKnowles]

Tagging #13208

Check this out @Keksmo
If doing this works

immich-machine-learning:
  image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
  volumes:
    - ./model-cache:/cache
  environment:
    - HF_XET_CACHE=/cache/huggingface-xet
  env_file:
    - .env
  restart: always

The above sets up a cache directory for the ML Container and tells the docker to download the ML image to the cache directory

Also before doing this clean and re-pull the image.

[Keksmo]

Tagging #13208

Check this out @Keksmo If doing this works

immich-machine-learning:
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
volumes:
- ./model-cache:/cache
environment:
- HF_XET_CACHE=/cache/huggingface-xet
env_file:
- .env
restart: always

The above sets up a cache directory for the ML Container and tells the docker to download the ML image to the cache directory

Also before doing this clean and re-pull the image.

Thank you for your answer, unfortunately it didn't change anything. The behavior is still the same (also I should note I am using rootless docker. I don't know if that's part of the problem, but to me the error log doesn't sound like it is)

[bo0tzz]

Just to double check, have you tried removing and re-pulling the container image?

[Keksmo]

Just to double check, have you tried removing and re-pulling the container image?

Yes just like my original post said, I removed and repulled the image, recreated the volume etc etc.

[Novapixel1010]

First is there reason you are doing this v1.141.1 instead of ${IMMICH_VERSION:-release}. Also there is now a stable release of immich as of yesterday. 2.0.0

Also are you mapping to ./model-cache or are you trying to use a docker volume? If so I pretty sure this is required.

volumes:
  model-cache:

This is a snippet from my working compose

  immich-machine-learning: 
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    environment:
      - MPLCONFIGDIR=/tmp/matplotlib
    volumes:
      - model-cache:/cache
    networks:
      - photo_net
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false

volumes:
  model-cache:

I am using portainer so my .env is change to stack.env Also to isolate I put it on a separate docker network photo_net It is not required so you can remove:

    networks:
      - photo_net

[Keksmo]

First is there reason you are doing this v1.141.1 instead of ${IMMICH_VERSION:-release}. Also there is now a stable release of immich as of yesterday. 2.0.0

Also are you mapping to ./model-cache or are you trying to use a docker volume? If so I pretty sure this is required.

volumes:
  model-cache:

This is a snippet from my working compose

  immich-machine-learning: 
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    environment:
      - MPLCONFIGDIR=/tmp/matplotlib
    volumes:
      - model-cache:/cache
    networks:
      - photo_net
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false

volumes:
  model-cache:

I am using portainer so my .env is change to stack.env Also to isolate I put it on a separate docker network photo_net It is not required so you can remove:

    networks:
      - photo_net

I got it pinned to that version because it is the most recent one that works, like I said, every newer version does not work. I've also tried your compose file and it didn't change anything unfortunately.