Skip to content

[NRT-873] Build the vendored bundle from uv.lock and guard Python 3.9 support - #1360

Merged
RisingOrange merged 8 commits into
mainfrom
worktree-py39-bundle-guard
Aug 27, 2026
Merged

[NRT-873] Build the vendored bundle from uv.lock and guard Python 3.9 support#1360
RisingOrange merged 8 commits into
mainfrom
worktree-py39-bundle-guard

Conversation

@RisingOrange

@RisingOrange RisingOrange commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Related issues

Proposed changes

scripts/build.py re-resolved the bundle group from pyproject.toml on every build. It never read uv.lock, and it evaluated markers against the build machine's Python (3.13) rather than the oldest interpreter the add-on has to run on — so any transitive that dropped Python 3.9 reached users unnoticed, which is what happened when asgiref 3.12 was released.

Anki bundles its own CPython. Official builds moved to 3.13 in 25.07, but the builds before that ship 3.9, and with a declared minimum Anki of 2.1.56 the add-on still has to run there. Roughly a quarter of active users are on an Anki predating 25.07.

The fix makes the break impossible rather than merely detected. The bundle now installs from the lockfile, in two layers, each with the oldest Python that can reach it: the bundle group with 3.9, bundle_modern (protobuf-py, which has required 3.10 since its first release) with 3.10. uv refuses a distribution declaring a newer floor, so the build fails instead of the add-on. They are separate dependency groups, so routing never reads markers.

Resolution only sees declared floors, so detection still covers the rest — an undeclared floor, a requirement a false marker dropped silently, a fault in assembling the archive — across the build, the packaged archive, and an import inside a real Anki on 3.9.

The build

  • From uv.lock, not re-resolved. uv export --locked --only-group <group> into uv pip install --no-deps. --locked also fails the build when the lockfile is out of step with pyproject.toml instead of quietly preferring one. Side effect: mashumaro is now pinned to the revision the lock recorded rather than tracking the fork's mutable master.
  • Every exported requirement must arrive. A requirement whose marker is false for the building interpreter or platform installs nothing and reports nothing, so its absence is the only signal there is. The build compares the export against the finished bundle's dist-info — catching a distribution that never arrived, not one whose files a prune removed.
  • Compiled extensions must be audited. abi3 fixes only the interpreter axis — protobuf-py-ext's wheel is cp310-abi3-manylinux_2_17_x86_64, unloadable on Windows, macOS or ARM. So what earns a place is the package falling back without it: protobuf-py catches the failed import (kept), nothing imports playhouse (its 2 MB dropped). Anything else fails the build.
  • Nothing survives a previous build. ankihub/lib is cleared and the previous .ankiaddon deleted — uv pip install --target and zip both update in place, so unvendored packages were surviving into local artifacts.
  • media_import installs with 3.9 too. Its requirements come from the submodule's own requirements.txt, not uv.lock, and were installed with whatever Python ran the build — outside every check, all of which were scoped to ankihub/lib. One pinned pure-Python package today, so nothing shipped was broken.

What gets checked

scripts/check_addon_import.py imports the built add-on the way Anki does — aqt.addons first, then ankihub.entry_point — and asserts where each vendored top-level package resolved from. The list comes from the bundle directory, and each assertion's direction follows from whether Anki had already imported that name, so on 2.1.56 certifi, typing_extensions and urllib3 must resolve outside the bundle and everything else inside it. It then renders a template through the add-on's own render_template_from_string(), covering packages nothing imports directly, such as sqlparse. It also parses every .py in the bundle under 3.9, reaching files nothing pulls in. Point it at a bundle carrying asgiref 3.12.1 and it reproduces the reported ImportError. It runs in the CI 3.9 lane, the only place with an Anki-on-3.9 environment.

scripts/release.sh adds three cheap checks over the extracted .ankiaddon: required members present, parses under 3.9, and the 3.10-only layer imports under 3.10. These live there because every workflow producing an artifact runs that script, so one added later inherits them — previously only the GitHub release was covered, not the copy users install from AnkiWeb. build.py is invoked as uv run --locked here and in setup-addon, since a plain sync or run re-locks a stale lockfile, after which build.py's own uv export --locked is satisfied by what it just wrote.

Why not the test suite. It runs the add-on on 3.9, but pytest plugins and factory-boy import django, asgiref and urllib3 from the dev environment before import ankihub prepends the bundle, so the bundled copies are shadowed and never exercised — that lane vendored the broken asgiref==3.12.1 and went green. Reordering conftest.py fixes it, but the guarantee would then be an import order the import sorter is free to undo, failing silently.

Smaller changes

  • Cap urllib3 in bundle. Not the Python floor the layers enforce — urllib3 2.x still supports 3.9 — but sentry-sdk 1.6.0 predates urllib3 2.0's API changes, and only the dev group's cap held it at 1.x.
  • Vendor tzdata unconditionally. Django declares it win32-only and the bundle is built on Linux, so the marker dropped it for exactly the users who need it. Latent until a template uses a date filter.
  • Django 4.2.27 → 4.2.30, the final 4.2 security release. Re-locking also moved asgiref, django-cotton 2.5.1→2.7.2, certifi, sqlparse and typing-extensions — already what a fresh resolve produced, but the lock is authoritative for the artifact for the first time, so the uv.lock hunks are worth a look.
  • Remove the sys.path entry for ankihub/lib/other — it has pointed at nothing since [INTP-141] Avoid comitting vendored dependencies #1176, while sitting ahead of the bundle.
  • README and justfile gain --group bundle_modern, without which a 3.10+ dev environment has no protobuf-py and mypy fails on ankihub/proto/*.py.
  • no_absolute_imports_from_ankihub skips scripts/ — the rule exists because Anki installs the add-on under its AnkiWeb id; build scripts are not shipped inside it.

How to reproduce

GOOGLE_API_KEY=dummy bash ./scripts/release.sh   # ends "... assembled, parses on Python 3.9, and its Python 3.10 layer loads"
xvfb-run -a uv run --exact --group aqt_legacy --python 3.9 python scripts/check_addon_import.py

The install-time refusal, which is what makes the original incident impossible:

uv pip install --python 3.9 --no-deps --target /tmp/x asgiref==3.12.1
# error: ... asgiref==3.12.1 depends on Python>=3.10 ... requirements are unsatisfiable

And detection, for a bundle that acquired a bad version some other way:

uv pip install --python 3.13 --no-deps --target ankihub/lib asgiref==3.12.1
xvfb-run -a uv run --exact --group aqt_legacy --python 3.9 python scripts/check_addon_import.py
# ImportError: cannot import name 'ParamSpec' from 'typing'

Also verified: the artifact holds 15 distributions (one more than before — tzdata) at 7.2 MB compressed; pytest ./tests/addon -m "not sequential and not performance" 731 passed; mypy clean on 118 files.

Verified by hand in Anki

The built artifact was installed and exercised in real Anki on three interpreters, since what changed is what the add-on ships:

  • Anki 25.02.7 / Python 3.9.23 — the configuration this exists for. Loads; django, asgiref.sync, peewee, structlog, sentry_sdk resolve from the bundle, urllib3 from Anki's venv. Sync and deck install work, so peewee runs without the stripped C extensions. django-cotton renders end to end — component loader, attribute expressions, nested components — the extent of what 2.5.1 → 2.7.2 could have broken.
  • Anki 25.09 / Python 3.13 / Linux — loads; urllib3 resolves from Anki's site-packages, so the shadowing contract holds in a real host; the abi3 extension loads.
  • Anki 25.09 / Python 3.13 / macOS arm64 — same Linux-built artifact. dlopen fails on the foreign x86_64 ELF, protobuf-py suppresses that ImportError and falls back to pure Python; messages still round-trip — the last known limit's fallback working with the wrong OS and architecture at once.

Further comments

Known limits, so nobody assumes more coverage than exists:

  • Prevention covers declared floors only. An undeclared one is caught by the 3.9 import check, or not at all.
  • That check imports the add-on; it does not call entry_point.run(), which needs a running Anki, and it imports the package as ankihub rather than under its AnkiWeb id.
  • It imports each package's top level, so a submodule is covered only if something reaches it. asgiref.sync — where the incident was — is covered because Django's template stack pulls it in, not because asgiref is vendored: import asgiref alone passes against a bundle carrying 3.12.1. The parse pass reaches every file but catches only syntax.
  • release.sh's parse pass narrows the publish-path gap rather than closing it, for the same reason.
  • The vendored protobuf extension is built for the build host; protobuf-py falls back where it cannot load, now confirmed on macOS arm64. Windows is untested, though it exercises the fallback more weakly — it ignores .so outright rather than attempting the load.

Known follow-ups, deliberately not here:

  • The AnkiWeb and S3 upload workflows are workflow_dispatch and have no CI gate, so a dispatch can publish without the 3.9 lane ever running. create_release.yml has the gate mechanism to copy.
  • Make that gate SHA-aware: create_release.yml checks the last CI run's conclusion but not its commit, so a push whose run has not been created yet slips through. Affects all regressions, not just 3.9 ones.
  • Build isolation is not locked. peewee is an sdist, so the same uv.lock can yield a different wheel later via a different build backend.
  • The Django render check renders plain HTML, so the cotton component loader — the novel vendored machinery — is not covered by it. It was verified by hand above.
  • Audit or bump sentry-sdk 1.6.0 (June 2022). It declares no Requires-Python and is what makes urllib3 a dependency at all.
  • certifi is still vendored though shadowed, for the same reason urllib3 is. A size cleanup, not a 3.9 fix.
  • create_release.yml builds the artifact before committing and tagging the version bump. Left alone deliberately: reordering would leave a pushed tag for a release that never shipped whenever a check fails.

@RisingOrange
RisingOrange force-pushed the worktree-py39-bundle-guard branch 2 times, most recently from 9fa86f1 to 1b26344 Compare August 5, 2026 11:56
@RisingOrange
RisingOrange marked this pull request as ready for review August 5, 2026 12:03
@RisingOrange
RisingOrange requested a review from a team August 5, 2026 12:03
… support

scripts/build.py re-resolved the bundle dependency group from pyproject.toml on
every build. It did not read uv.lock, and it evaluated markers against the build
machine's Python rather than the oldest interpreter the add-on has to run on, so
any transitive that dropped Python 3.9 support reached users unnoticed - which is
what happened when asgiref 3.12 was released.

Anki bundles its own CPython. Official builds moved to 3.13 in 25.07, but 3.9
remained the minimum supported version through 25.09 and the builds before 25.07
do ship it, so with a declared minimum Anki of 2.1.56 the add-on still has to run
there. Roughly a quarter of active add-on users are on an Anki predating 25.07.

Install from uv.lock instead, in two layers, each with the oldest Python that can
reach it: the bundle group with 3.9, and bundle_modern - protobuf-py, which has
required 3.10 since its first release - with 3.10. uv then refuses a distribution
declaring a newer floor, so the build fails rather than the add-on. The layers are
dependency groups, so nothing routes on markers.

Assert every exported requirement produced a distribution: a requirement whose
marker is false for the building interpreter or platform installs nothing and
reports nothing, so its absence is the only signal there is. This reads dist-info,
so what it catches is a distribution that never arrived, not one whose files a
prune removed while leaving its metadata. Assert the layers are disjoint, since they install into one directory in
sequence. Clear ankihub/lib before installing and delete the previous .ankiaddon
before re-zipping - uv pip install --target and zip both update in place.

Vendor tzdata unconditionally. Django declares it win32-only, the build runs on
Linux, and one artifact ships to every platform, so the marker dropped it for the
only users who need it. Bump Django to 4.2.30, the last 4.2 security release.
Drop playhouse's compiled extensions, which cannot load off the build host.
Remove the sys.path entry for ankihub/lib/other, which has pointed at nothing
since vendored dependencies stopped being committed.

Resolution covers declared floors only, so scripts/check_addon_import.py imports
the built add-on the way Anki does - aqt.addons first, then ankihub.entry_point -
checks where each of the bundle's modules resolved from, and renders a template
through the add-on's own Django setup. It runs in the CI 3.9 lane, the only place
with an Anki-on-3.9 environment. The test suite cannot do this even though it runs
on 3.9: pytest plugins and factory-boy import django, asgiref and urllib3 from the
development environment before `import ankihub` prepends the bundle, so the
bundled copies are shadowed there and never exercised.

scripts/release.sh adds two checks over the extracted archive - required members
present, and the 3.10-only layer importing under 3.10. Every workflow producing an
artifact runs that script, so the AnkiWeb and S3 uploads and the PR test builds
inherit them rather than needing their own.
@RisingOrange
RisingOrange force-pushed the worktree-py39-bundle-guard branch from 1b26344 to 431ae7f Compare August 5, 2026 12:11
@cacoze
cacoze requested a review from abdnh August 18, 2026 20:13
RisingOrange and others added 6 commits August 26, 2026 11:09
…t on 3.9

The prune named playhouse, but the hazard is not playhouse's: one artifact ships
to every interpreter and platform Anki runs on, and the layered install compiles
for 3.9 or 3.10 rather than the build machine's Python, so any extension that is
not abi3 can only load by accident. Sweep the bundle instead - abi3 passes,
playhouse's optional ones are dropped, anything else fails the build rather than
being deleted quietly, since something may need it to import.

check_addon_import.py reaches a module only if something imports it, so its
coverage of a submodule is whatever the add-on's own import graph happens to
touch: `asgiref.sync`, the module the incident was in, is covered because
Django's template stack pulls it in, not because asgiref is vendored - `import
asgiref` alone passes against a bundle carrying 3.12.1. Parse every .py in the
bundle under 3.9 as well. That reaches every file, but it is the weaker signal
and not what this rests on: it catches syntax a release started using, not the
`from typing import ParamSpec` break, which parses on 3.9 and fails on import.

Don't let an option line uv emits, such as --index-url, parse as a requirement
named "-index-url" and fail the bundle-contents check with a confusing message.

Unlink ankihub/lib when it is a symlink: worktrees set up for running Anki point
it at the main checkout's copy, where rmtree raises an errno-less OSError and
installing would write to the other checkout.

Also check VERSION is in the archive, and record why the vendored urllib3 is 1.x
- the dev group caps it, so nothing in the bundle group shows the reason.
abi3 answers only half the question. protobuf-py-ext's wheel is
cp310-abi3-manylinux_2_17_x86_64: stable across CPython versions, and still
unable to load on Windows, macOS or ARM. Letting the tag alone grant a pass
would have waved through a future dependency whose extension is required,
breaking every platform except the Linux builder, with no check firing.

What makes an extension safe to vendor into a single cross-platform artifact is
the package treating it as optional. Both of these do, so list them and fail on
anything else: protobuf-py catches the failed import and falls back to pure
Python, so its extension is kept for the platform it does load on, and nothing
imports playhouse at all, so its 2 MB are dropped.

Refuse a line in uv's export that is neither a comment, a --hash continuation,
nor an option, instead of skipping it. Options add no distribution and stay
skipped; an editable would, and silently leaves it outside every later check.

Both found by Codex reviewing the branch.
The Python 3.9 import check needs an Anki running on 3.9, so it exists only in
the CI lane that has one. But the AnkiWeb and S3 upload workflows are
workflow_dispatch and call this script directly, so an artifact can be published
without any 3.9 check having run. Parse the extracted bundle under 3.9 here.
That narrows the gap rather than closing it: it catches syntax a release started
using, not a 3.10-only import, which is the shape the original incident had.

Invoke build.py as `uv run --locked`. A plain `uv run` syncs first and will
re-lock a lockfile that has fallen behind pyproject.toml, after which build.py's
own `uv export --locked` sees the lock it just wrote and is satisfied - so the
artifact could be built from a resolution nobody reviewed.

Delete temporary files with -exec rather than a pipe into xargs, which splits on
whitespace: a vendored path containing a space became two unrelated arguments to
`rm -rf`. -depth so a directory goes only after find has walked into it, and an
escaped dot in the .pyc pattern so it cannot match a character it did not mean.

Found by Codex reviewing the branch.
Cut what restated the code: a comment naming what OLDEST_MODERN_PYTHON already
says, a docstring repeating its function's name, one announcing that load_addon
loads the add-on. Compressed the rest to the fact a future editor could break -
what uv export does with --locked, that uv's own output is not re-parsable, that
a false marker installs nothing and reports nothing, that zip updates in place.

Moved the note about the vendored urllib3 being held at 1.x to the dev group's
own cap, which is what holds it. Attached to sentry-sdk in the bundle group it
never reached the person who could loosen it.

Dropped from the extension audit the platform it was confirmed on and the
argument for failing rather than guessing - the commit and the error message
carry those.
…all sites

media_import ships inside the same artifact and reaches the same interpreters,
but its requirements were installed with whatever Python ran the build - 3.13 in
every workflow - so nothing refused a distribution that had dropped 3.9. They
are not in uv.lock either; the submodule carries its own requirements.txt, and
none of the checks reached them: the extension audit and the bundle-contents
check are scoped to ankihub/lib, and so was the archive's parse pass. Install
them with the oldest interpreter too, and parse them with the rest. Today the
file holds one pinned pure-Python package, so nothing shipped was broken.

Pass --locked at the two remaining call sites. setup-addon syncs and then runs
build.py without it, so a stale lockfile is re-locked in passing and the CI lane
named for checking Python 3.9 attests to a bundle release.sh would not build.

Cap urllib3 in the bundle group rather than leaving it to the dev group's cap.
This is not the Python floor the layers enforce - urllib3 2.x still supports 3.9
- but sentry-sdk 1.6.0 predates urllib3 2.0's API changes, and the bundled copy
is never import-tested, since Anki preloads its own and the origin check
requires the vendored one to lose. Re-locking adds the requirement and moves no
versions.

Remove the artifact when a check fails. It is zipped before it is checked, so a
failure left a freshly-timestamped broken one at the repository root.

All four found by Fable reviewing the branch.
@RisingOrange
RisingOrange merged commit 68a817e into main Aug 27, 2026
8 checks passed
@RisingOrange
RisingOrange deleted the worktree-py39-bundle-guard branch August 27, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants