fix(docker): align the API image's Python with the rest of the stack - #171
Merged
Conversation
The API container ran Python 3.14 while all three worker lanes, every CI job, mypy, and the documented prerequisite were on 3.12. Nothing was broken by it, but the interpreter serving every HTTP request was the only one the test suite never ran on. The worker's Python is not a free choice: it arrives with the PyTorch base image, which is held at 2.11.x because torchaudio has published nothing above it, and that image ships 3.12. The alignment therefore has to go downward. Both stages of Dockerfile.api move to python:3.12-slim, pinned by digest. Moving one stage alone would build the venv on one interpreter and run it on another, which can still appear to work. The root cause was a gap in .github/dependabot.yml. Its docker ecosystem block held pytorch/pytorch but said nothing about python, so Dependabot walked the API base from 3.12 to 3.14 unreviewed and no gate noticed. A matching hold now covers python, and check_held_pins.py grows a third check to back it up: EXPECTED_PYTHON records the minor the held base ships, and PYTHON_DECLARATIONS asserts that the Dockerfile (both stages), the workflows, the mypy target, and the two documented prerequisites all agree with it. The check is offline, so it gates every pull request alongside the existing drift check, and it fails a half-applied bump as well as a wholesale one. Node majors are held for the same reason, and the comment above the worker-io Node stage is corrected. It still described node:22 after Dependabot moved the tag to 26, and it did not record why the two Node bases differ. They are not interchangeable: worker-io copies the bare node binary into the Ubuntu PyTorch base, so its donor must be glibc-linked, while the frontend runs Node standalone and takes the smaller musl/Alpine image. An Alpine build would find no loader in the worker base at all. Verified by building the image rather than by reading the Dockerfile. The runtime, the venv interpreter, and the venv site-packages directory all report 3.12.13; the compiled dependencies (numpy, google-re2, reportlab, markdown_pdf) import; and Trivy at the release gate's settings reports no fixable CRITICAL or HIGH findings. Refs #169
Valtora
added a commit
that referenced
this pull request
Jul 30, 2026
Bump docs/VERSION to 2.2.0 so the tag validates, and fill in the release-notes template for this range instead of hand-editing the published body afterwards, which is how v2.1.0 was done. A minor bump rather than a patch: #168 adds a capture action that did not exist before, stopping and processing a paused recording. Refs: #162, #163, #165, #167, #168, #169, #170, #171
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
The
nojoin-apicontainer ran Python 3.14 while all three worker lanes, every CI job, mypy, and the documented prerequisite were on 3.12. Nothing was broken by it, but the interpreter serving every HTTP request was the only one the test suite never ran on. Measured on the live deployment before the change:nojoin-apinojoin-worker-gpu/-cpu/-ioThe alignment goes downward because the worker's Python is not a free choice: it arrives with the PyTorch base image, held at 2.11.x because torchaudio has published nothing above it, and that image ships 3.12. Raising everything to 3.14 would require breaking that hold, which is outside this repository's control.
Root cause. The
dockerecosystem block in.github/dependabot.ymlheldpytorch/pytorchbut said nothing aboutpython, so Dependabot walked the API base from 3.12 to 3.14 unreviewed and no gate noticed.What changed
docker/Dockerfile.api— both stages move topython:3.12-slim, pinned by digest. Both together: moving one alone builds the venv on one interpreter and runs it on another, which can still appear to work..github/dependabot.yml— holds added forpython(>=3.13.0) andnode(>=27.0.0).scripts/check_held_pins.py— a third check alongside the existing drift and release checks.EXPECTED_PYTHONrecords the minor the held base ships;PYTHON_DECLARATIONSasserts the Dockerfile (both stages), the workflows, the mypy target, and both documented prerequisites all agree. Offline, so it gates every PR.docker/Dockerfile.worker-io— comment corrected. It still describednode:22after Dependabot moved the tag to 26, and did not record why the two Node bases differ.docs/DEVELOPMENT.md— the held-pins policy gains the step this PR implements: encode what a hold transitively fixes, not just the pin.No new dependencies.
Type of change
Checks run
source .venv/bin/activate && pytest— 1176 passedpython scripts/check.py— all nine checks pass, exit 0python3 scripts/validate_docs.py— 20 filespython3 scripts/validate_alembic.py— heade1a7c93b8d24Migration impact
Documentation impact
docs/DEVELOPMENT.md"Held Pins and Unfixable Advisories" gains a step for constraints a hold transitively fixes. The stated prerequisite stays Python 3.12 — after this change every container, CI job, and mypy target is 3.12, so the existing wording becomes accurate rather than aspirational and needed no edit.Security impact
A base-image change alters the vulnerability surface, so the rebuilt image was scanned at the release gate's exact settings (
--severity CRITICAL,HIGH --ignore-unfixed --ignorefile .trivyignore --exit-code 1): 0 findings, exit code 0. No.trivyignoreentries were added or needed.python:3.12-slimis Debian 13 (trixie), the same distro generation as the 3.14 image it replaces, so there is no OS-package regression.Manual verification
The interpreter was verified by building the image, not by reading the Dockerfile.
Compiled dependencies import cleanly on 3.12 —
numpy 2.4.6,google-re2,reportlab 5.0.0,markdown_pdf— which is the risk a lockfile read cannot cover.The guard was demonstrated against the bad cases, not just added. Each was applied to a working tree, checked, and reverted:
FROMdeclares the Python version 1 time(s), expected 2python-versionmoved to 3.13python_versionmoved to 3.14The last row is the one that matters for durability: losing coverage fails loudly rather than turning into a green tick.
On the two Node bases — investigated for possible unification; they must stay different, and this is now recorded in the Dockerfile:
docker/Dockerfile.worker-iousesnode:26-bookworm-slimpurely as a donor stage — the barenodebinary is copied into the PyTorch base, which is Ubuntu 24.04 / glibc 2.39. Bookworm builds against glibc 2.36, and glibc is forward compatible, so it runs there. Verified:lddshowslibc.so.6.frontend/Dockerfileusesnode:26-alpine, which links against musl (/lib/ld-musl-x86_64.so.1, verified). It runs Node standalone, so the smaller image is correct.Unifying toward Alpine would break worker-io outright — a musl binary has no loader in the Ubuntu base. Unifying toward bookworm would be a pure size regression for the frontend. Both are held against unreviewed major bumps instead.
Screenshots (if relevant)
Not applicable.