Speed up CI: cache scoping, Python test staging, and Morango/postgres test overheads - #15167
Open
rtibbles wants to merge 9 commits into
Open
Speed up CI: cache scoping, Python test staging, and Morango/postgres test overheads#15167rtibbles wants to merge 9 commits into
rtibbles wants to merge 9 commits into
Conversation
Every stage 2 job waited on stage 1, and stage 1 waited on postgres. Postgres runs the same suite roughly three times slower than sqlite (23.6 vs 8.0 minutes measured), so a run that could have started its stage 2 jobs at 8 minutes started them at 24, and finished at 44. Gate stage 2 on postgres instead. It still starts off pre_job, so it now runs alongside stage 2 rather than in front of it, and the required Python tests check still covers it. Expected wall clock ~27 minutes. The tradeoff is that a postgres failure no longer short-circuits stage 2, so a PR breaking only postgres spends that compute before going red. Drop max-parallel: 5 while here. It held the 3.14 leg back by 5.5 minutes on the stage 2 matrix; on the other three matrices it is below the cap and never did anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The source image is an input to the base image and to nothing else — kolibri.Pifile reads base.img and the debs. Caching both stored the same work twice, at ~1.5GB per run. Keep the base image, which is the last artifact that does not depend on the deb. Dropping the source cache means make must not go looking for source.img when a base image is already present: it is a missing prerequisite, so make would re-download 500MB and then rebuild the base image it already had. Guard on the base image instead, and give images/base.img the real prerequisite for the case where it is absent. pimod goes with it. A depth-1 clone of a pinned tag is 0.56s and 360KB, it is needed on every run whatever the cache state, and it shared the source key, so any Makefile edit re-cloned it anyway. Add the Makefile to the base image key, since it pins SOURCE_URL. Both old keys omitted it, so bumping the raspios version restored a base image built from the previous one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cache is readable only from the ref that wrote it or from the default branch, and fork PRs cannot write caches at all, so develop is the only scope that serves every PR. Warming just the WHL build left the rest of the graph writing caches nowhere useful. The Pi image is why this matters. Its keys are exact-match with no restore-keys, so a PR that misses rebuilds the base image and saves it under refs/pull/N/merge, where no other PR can read it. Four in-flight PRs were holding 5.95GB that way — 61% of the repo's 10GB budget — and had evicted the develop-scoped webpack cache that build_whl.yml exists to prime, so PRs were also paying for cold webpack compiles. Priming on develop turns those misses into hits, and actions/cache skips its save step on a hit, so the per-PR copies stop being written. warm_build_cache.yml is now a subset of this workflow, so drop it. pr_build_comment.yml needs no change: it already notices when no open PR matches the head SHA and skips. Concurrency falls back to ref rather than run_id, since head_ref is empty on push and successive develop pushes would otherwise build concurrently and race on the same keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Morango picks a key implementation at import: Cryptography, then M2Crypto, then a pure-Python fallback. Nothing pulled cryptography into the test environment, so it took the fallback and spent 3.4s generating a 2048-bit keypair for every facility root certificate. With it, the same keygen is 0.098s - a 34x difference, measured by profiling generateuserdata. This moves tests towards production rather than away from it: shipped Kolibri gets cryptography from requirements/cext.txt, so the fallback was only ever exercised in development. A floor rather than a pin, so 3.6 and 3.7 resolve to 40.0.2, their last supported release, without pinning any newer Python to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three overheads, none of which are what these tests exercise. Every multiple_kolibri_servers entry ran a full migration to build the home its servers are copied from - 19s, 15 times over. The home depends only on the kwargs it was built with and is never written to afterwards, so build one per distinct set and reuse it. KOLIBRI_TEST_PRESEEDED_HOME already existed for this but was read from nowhere, so it never took effect; it now overrides the location rather than being the only way in. The spawned servers ran base settings, so they hashed with Django's PBKDF2 default while the test process used MD5 - which is why the MD5 tweak was disabled for integration runs, an md5$ hash being unverifiable on the server. Give the servers a settings module that inherits base and agrees on the hasher, and both sides can use it: pbkdf2_hmac was 1.2s per generateuserdata alone. Polling server startup every 0.5s spends a quarter second per start, on average, after the server is already answering. Together these take SingleUserSyncRegressionsTestCase from 325s to 210s, alongside the cryptography change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every create_model is its own kolibri shell startup, and _create_objects made seven of them per call, ten times per run. It also read each object back through the server's db alias purely to get an id to pass to the next create. run_scenario sends a function reference instead, so a whole group of related objects costs one startup and the intermediate lookups become local variables. Creation still happens in the server's own process through the normal ORM path, which is what the module docstring requires and what Morango's save() hooks depend on - a fixture would not, since loaddata saves raw and skips them. test_chaos_sync goes from 305.5s to 283.8s. The scenarios live in a module rather than the test file because the server has to import them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every COMMIT against the service container costs an fsync, and the test suite makes a lot of them: profiling 18 content tests showed 1565 commits at 1.9ms each, with the process at 20% CPU. It was waiting on the disk, not computing. The database is created and thrown away within the job, so there is nothing for that durability to protect. The content suite, 1424 tests, goes from 505s to 158s. The same 3.2x shows up on test_annotation.py alone (81s to 25s), so it is the commit path rather than anything specific to one module. Not applied to the SSL smoke test, whose test step is 14s - the rest of that job is dependency install and container startup. actionlint's schema has no `command:` key for services, though the runner honors it; .github/actionlint.yaml already ignored that false positive for tox.yml, so extend it to the two workflows that now need it as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
DEV: dev-ops
Contributor
Build Artifacts
Smoke test screenshot |
setup-uv builds its cache key from the runner's system Python, not the matrix version, so every leg of every matrix shared one key and raced to reserve it. 52 of the 88 warnings in a PR run were losers of that race. Per-version keys would end the race but add ~400MB to a repo cache pool that is already over cap and evicting. Not worth it: on a verified cache miss, downloading 111 packages and building kolibri and three sdists took 2.46s in total, and 860ms on the 3.6 buster container. Building kolibri dominates the step at 30-60s, and uv cannot cache that. cache-python was also set on four jobs that never install a managed Python, and the WHL test matrix checks out nothing, so its key could never invalidate and its empty workdir warned on every leg. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013xxxJ57Fkk7mbU6kKLNrva
rtibbles
force-pushed
the
ci-cache-scoping-and-test-staging
branch
5 times, most recently
from
August 9, 2026 04:11
b0423ff to
2e16095
Compare
Its file finder enumerates the tree by streaming a full `git archive HEAD` and parsing the tar — 69MB of blob content here to list 5378 names. Cold on a runner that is 25-44s, and 35 jobs in a PR run install the project, so it costs ~23 minutes of runner time per push. Warm it is 0.2s, which is why it never shows up locally. SETUPTOOLS_SCM_IGNORE_VCS_ROOTS short-circuits the finder before the archive call. Versioning is a separate code path and is unaffected. There is no pyproject.toml equivalent — the finder is a global setuptools entry point that never sees our config. Measured across the Python matrix, installs go from 31-55s to 3-6s. Windows sets the variable from `git rev-parse --show-toplevel` instead of github.workspace: setuptools_scm compares the ignored roots to git's own output as strings, and github.workspace is backslashed there. The buster containers neither set it nor need it — they install with `uv pip install --system` and run from PYTHONPATH, so they never build the project. MANIFEST.in becomes the sole authority, which is how it already reads. That drops 2074 files and 13.5MB of repo furniture the finder was force-including into the sdist on top of it: a 323KB nssm.exe, docs screenshots, uv.lock and pnpm-lock.yaml, eslint/jest/prettier config, .github, packages, platforms. Under kolibri/ it takes the locale .po files too, so those are now declared; the LC_MESSAGES README.md and .tbx glossary go, being neither declared nor wanted. Wheel contents are otherwise byte-identical, and the debian package, which is built from the sdist, still builds and serves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013xxxJ57Fkk7mbU6kKLNrva
rtibbles
force-pushed
the
ci-cache-scoping-and-test-staging
branch
from
August 9, 2026 04:46
2e16095 to
91de108
Compare
rtibbles
marked this pull request as ready for review
August 10, 2026 18:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Attempt to optimize CI runs on PRs:
Measured effect
Medians over CI runs. Baseline is contemporaneous runs of the unmodified workflow (7 other PRs plus 2 develop pushes, 8-9 Aug); after is 7 runs on this branch.
Spread matters as much as the median: the postgres test step was 560-1133 s and is now 537-572 s, and the slowest baseline run (46.4 min) was slow because postgres took 20.6 min. Runner time only drops on runs including the setuptools_scm commit; everything else here moves wall clock, not runner time.
References
Prior caching work: #14852.
Reviewer guidance
kolibri/deployment/default/settings/test.py:28— MD5 hashing now applies to integration runs too, and only works because the spawned servers get the matching hasher fromsettings/integration_test_server.py. Check nothing else spawns a server on base settings and then verifies a password.kolibri/core/auth/test/sync_utils.py:193— the preseeded home is now built once and copied per server, so anything that writes back to it would leak between tests rather than being rebuilt..github/workflows/pr_build_kolibri.yml:8— the concurrency group falls back togithub.refon push; confirm that cannot cancel a PR run that shares a group.AI usage
Used Claude Code to profile the test suites, identify the bottlenecks from CI logs and cProfile, and implement the changes. Verified with the Morango integration suite, the content test suite under both database engines, and prek.