Skip to content

fix(docker): align the API image's Python with the rest of the stack - #171

Merged
Valtora merged 1 commit into
mainfrom
fix/align-api-python-version
Jul 30, 2026
Merged

fix(docker): align the API image's Python with the rest of the stack#171
Valtora merged 1 commit into
mainfrom
fix/align-api-python-version

Conversation

@Valtora

@Valtora Valtora commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Pull Request

Description

The nojoin-api container ran Python 3.14 while all three worker lanes, every CI job, mypy, and the documented prerequisite were on 3.12. Nothing was broken by it, but the interpreter serving every HTTP request was the only one the test suite never ran on. Measured on the live deployment before the change:

Surface Before After
nojoin-api 3.14.6 3.12.13
nojoin-worker-gpu / -cpu / -io 3.12.3 3.12.3 (unchanged)
CI (11 sites), mypy, docs 3.12 3.12 (unchanged)

The alignment goes downward because the worker's Python is not a free choice: it arrives with the PyTorch base image, held at 2.11.x because torchaudio has published nothing above it, and that image ships 3.12. Raising everything to 3.14 would require breaking that hold, which is outside this repository's control.

Root cause. The docker ecosystem block in .github/dependabot.yml held pytorch/pytorch but said nothing about python, so Dependabot walked the API base from 3.12 to 3.14 unreviewed and no gate noticed.

What changed

  • docker/Dockerfile.api — both stages move to python:3.12-slim, pinned by digest. Both together: moving one alone builds the venv on one interpreter and runs it on another, which can still appear to work.
  • .github/dependabot.yml — holds added for python (>=3.13.0) and node (>=27.0.0).
  • scripts/check_held_pins.py — a third check alongside the existing drift and release checks. EXPECTED_PYTHON records the minor the held base ships; PYTHON_DECLARATIONS asserts the Dockerfile (both stages), the workflows, the mypy target, and both documented prerequisites all agree. Offline, so it gates every PR.
  • docker/Dockerfile.worker-io — comment corrected. It still described node:22 after Dependabot moved the tag to 26, and did not record why the two Node bases differ.
  • docs/DEVELOPMENT.md — the held-pins policy gains the step this PR implements: encode what a hold transitively fixes, not just the pin.

No new dependencies.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behaviour)
  • Documentation update

Checks run

  • Backend tests: source .venv/bin/activate && pytest — 1176 passed
  • Python quality: python scripts/check.py — all nine checks pass, exit 0
  • Frontend lint: not touched
  • Frontend unit tests: not touched
  • Frontend build: not touched
  • Docs validation: python3 scripts/validate_docs.py — 20 files
  • Alembic validation: python3 scripts/validate_alembic.py — head e1a7c93b8d24

Migration impact

  • No database migration in this PR.
  • Adds an Alembic migration.

Documentation impact

  • No documentation change required.
  • Updated the relevant guide(s) in the same PR.

docs/DEVELOPMENT.md "Held Pins and Unfixable Advisories" gains a step for constraints a hold transitively fixes. The stated prerequisite stays Python 3.12 — after this change every container, CI job, and mypy target is 3.12, so the existing wording becomes accurate rather than aspirational and needed no edit.

Security impact

  • No security-sensitive change.
  • Touches auth, tokens, encryption, capture ownership, or exposure.

A base-image change alters the vulnerability surface, so the rebuilt image was scanned at the release gate's exact settings (--severity CRITICAL,HIGH --ignore-unfixed --ignorefile .trivyignore --exit-code 1): 0 findings, exit code 0. No .trivyignore entries were added or needed. python:3.12-slim is Debian 13 (trixie), the same distro generation as the 3.14 image it replaces, so there is no OS-package regression.

Manual verification

The interpreter was verified by building the image, not by reading the Dockerfile.

$ docker run --rm --entrypoint python nojoin-api:py312-verify -c "import sys; print(sys.version)"
3.12.13 (main, Jul 14 2026, 02:09:00) [GCC 14.2.0]

$ docker run --rm --entrypoint /opt/venv/bin/python nojoin-api:py312-verify -V
Python 3.12.13                      # venv interpreter matches the runtime

$ docker run --rm --entrypoint sh nojoin-api:py312-verify -c 'ls -d /opt/venv/lib/python*'
/opt/venv/lib/python3.12            # proves which minor built the venv

Compiled dependencies import cleanly on 3.12 — numpy 2.4.6, google-re2, reportlab 5.0.0, markdown_pdf — which is the risk a lockfile read cannot cover.

The guard was demonstrated against the bad cases, not just added. Each was applied to a working tree, checked, and reverted:

Injected fault Result
Both stages walked back to 3.14 (the original bug) caught, 2 drift lines, exit 1
Only the builder stage bumped (the subtle one) caught, exit 1
A build stage deleted, leaving one FROM caught by the count assertion: declares the Python version 1 time(s), expected 2
CI python-version moved to 3.13 caught at every site, exit 1
mypy python_version moved to 3.14 caught, exit 1
Docs prerequisite reworded so the regex stops matching caught as unchecked, not silently passed

The last row is the one that matters for durability: losing coverage fails loudly rather than turning into a green tick.

On the two Node bases — investigated for possible unification; they must stay different, and this is now recorded in the Dockerfile:

  • docker/Dockerfile.worker-io uses node:26-bookworm-slim purely as a donor stage — the bare node binary is copied into the PyTorch base, which is Ubuntu 24.04 / glibc 2.39. Bookworm builds against glibc 2.36, and glibc is forward compatible, so it runs there. Verified: ldd shows libc.so.6.
  • frontend/Dockerfile uses node:26-alpine, which links against musl (/lib/ld-musl-x86_64.so.1, verified). It runs Node standalone, so the smaller image is correct.

Unifying toward Alpine would break worker-io outright — a musl binary has no loader in the Ubuntu base. Unifying toward bookworm would be a pure size regression for the frontend. Both are held against unreviewed major bumps instead.

Screenshots (if relevant)

Not applicable.

The API container ran Python 3.14 while all three worker lanes, every CI
job, mypy, and the documented prerequisite were on 3.12. Nothing was
broken by it, but the interpreter serving every HTTP request was the only
one the test suite never ran on.

The worker's Python is not a free choice: it arrives with the PyTorch base
image, which is held at 2.11.x because torchaudio has published nothing
above it, and that image ships 3.12. The alignment therefore has to go
downward. Both stages of Dockerfile.api move to python:3.12-slim, pinned
by digest. Moving one stage alone would build the venv on one interpreter
and run it on another, which can still appear to work.

The root cause was a gap in .github/dependabot.yml. Its docker ecosystem
block held pytorch/pytorch but said nothing about python, so Dependabot
walked the API base from 3.12 to 3.14 unreviewed and no gate noticed. A
matching hold now covers python, and check_held_pins.py grows a third
check to back it up: EXPECTED_PYTHON records the minor the held base
ships, and PYTHON_DECLARATIONS asserts that the Dockerfile (both stages),
the workflows, the mypy target, and the two documented prerequisites all
agree with it. The check is offline, so it gates every pull request
alongside the existing drift check, and it fails a half-applied bump as
well as a wholesale one.

Node majors are held for the same reason, and the comment above the
worker-io Node stage is corrected. It still described node:22 after
Dependabot moved the tag to 26, and it did not record why the two Node
bases differ. They are not interchangeable: worker-io copies the bare node
binary into the Ubuntu PyTorch base, so its donor must be glibc-linked,
while the frontend runs Node standalone and takes the smaller musl/Alpine
image. An Alpine build would find no loader in the worker base at all.

Verified by building the image rather than by reading the Dockerfile. The
runtime, the venv interpreter, and the venv site-packages directory all
report 3.12.13; the compiled dependencies (numpy, google-re2, reportlab,
markdown_pdf) import; and Trivy at the release gate's settings reports no
fixable CRITICAL or HIGH findings.

Refs #169
@Valtora
Valtora merged commit b6d81e0 into main Jul 30, 2026
19 of 20 checks passed
@Valtora
Valtora deleted the fix/align-api-python-version branch July 30, 2026 00:00
Valtora added a commit that referenced this pull request Jul 30, 2026
Bump docs/VERSION to 2.2.0 so the tag validates, and fill in the
release-notes template for this range instead of hand-editing the
published body afterwards, which is how v2.1.0 was done.

A minor bump rather than a patch: #168 adds a capture action that did
not exist before, stopping and processing a paused recording.

Refs: #162, #163, #165, #167, #168, #169, #170, #171
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant