fix(Cache): give each cache process its own temporary file names - #42752
Open
kim-em wants to merge 1 commit into
Open
fix(Cache): give each cache process its own temporary file names#42752kim-em wants to merge 1 commit into
kim-em wants to merge 1 commit into
Conversation
The cache directory is per user, not per checkout, so several `cache` runs share it: different projects, several worktrees of one project, or a fleet of agents on one machine. Both temporary paths inside it were fixed names, so those runs wrote each other's files. `curl.cfg` is written, handed to curl, and deleted. When a second run overwrote it in the window before curl read it, the first run downloaded the second's file list, then reported its own files as missing and rebuilt them; whichever run finished second also hit an uncaught exception deleting a file that was no longer there. `<hash>.ltar.part` is worse: two curls writing one path interleave, and the winner of the rename leaves a corrupt `.ltar` that every later run finds, trusts, and fails to decompress. Name both after the writing process, so a run only ever renames or removes its own files. `hashFromFileName` keeps reading untagged `.ltar.part` names, which a cache directory written by an older version may still hold.
PR summary 50b5b91641Import changes for modified filesNo significant changes to the import graph Import changes for all files
|
kim-em
added a commit
to kim-em/TauCetiWorker
that referenced
this pull request
Aug 14, 2026
From a review of the first draft, which pointed every worker's MATHLIB_CACHE_DIR at one directory. That reintroduced a race the per-worker homes had been hiding: `lake exe cache get` takes no lock, and until every checkout runs a Mathlib carrying leanprover-community/mathlib4#42752, two concurrent runs share `curl.cfg` and share each `<hash>.ltar.part`. The outcome is not a failed round but a corrupt `.ltar` under a name every later run trusts, for every worker and the operator alike — and the risk peaks exactly when the pool pays off most, just after a bump, when several workers fetch the same 8,600 files at once. Each worker now downloads into its own directory and exchanges finished files with the pool by hardlink before the agent starts: it promotes what it fetched last round, then takes a link to everything the pool has that it lacks. Only complete artifacts reach the pool, an existing name is never replaced, and a link is not a copy, so the download saving stands and the disk saving with it. Toolchains are still shared outright, since an elan install takes a per-toolchain lock and lands by rename. LAKE_CACHE_DIR is now pinned per-worker rather than following ELAN_HOME under the toolchain directory: it is written throughout a build, not once at install, and that also keeps a one-worker LAKE_ARTIFACT_CACHE experiment honest. The migration script verifies before it deletes. Equal name plus equal size was not the proof the first draft claimed: a `.ltar` is named by a hash of build inputs rather than of its bytes, so pruning now requires the same inode, the same symlink target, or an actual byte comparison, deletes the verified paths one by one instead of removing a tree that also holds files it never examined, and leaves Lake's mutable per-toolchain store out of the pool entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rt6ZAuD5HvTVj1GVexU6CG
kim-em
added a commit
to kim-em/TauCetiWorker
that referenced
this pull request
Aug 14, 2026
* fix: pool the Lean build caches across workers Mathlib's cache tool resolves ~/.cache/mathlib and elan resolves ~/.elan through $HOME, so the per-worker $HOME that isolate_home() creates for credentials also gave every worker a private copy of the same public, content-addressed downloads. On a five-worker fleet that cost half of one week's 10.2 GB of .ltar traffic to Mathlib's cache server, and 22 toolchain installs covering 6 distinct toolchains. Export MATHLIB_CACHE_DIR and ELAN_HOME from both isolate_home() paths, resolved through _host_home() so the already-isolated path a round child runs computes the same pool without waiting for a fleet restart. An operator-set value still wins. Lake's artifact cache lives under the toolchain directory, so it follows ELAN_HOME; the bubble environment drops both, since the container is configured by Bubble's own cache proxy. scripts/share-build-caches folds pre-existing per-worker copies into the pool by hardlink, and prunes them only where the pool already holds the same path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rt6ZAuD5HvTVj1GVexU6CG * style: ruff format the new test and script * fix: pool the Mathlib cache by hardlink instead of by shared directory From a review of the first draft, which pointed every worker's MATHLIB_CACHE_DIR at one directory. That reintroduced a race the per-worker homes had been hiding: `lake exe cache get` takes no lock, and until every checkout runs a Mathlib carrying leanprover-community/mathlib4#42752, two concurrent runs share `curl.cfg` and share each `<hash>.ltar.part`. The outcome is not a failed round but a corrupt `.ltar` under a name every later run trusts, for every worker and the operator alike — and the risk peaks exactly when the pool pays off most, just after a bump, when several workers fetch the same 8,600 files at once. Each worker now downloads into its own directory and exchanges finished files with the pool by hardlink before the agent starts: it promotes what it fetched last round, then takes a link to everything the pool has that it lacks. Only complete artifacts reach the pool, an existing name is never replaced, and a link is not a copy, so the download saving stands and the disk saving with it. Toolchains are still shared outright, since an elan install takes a per-toolchain lock and lands by rename. LAKE_CACHE_DIR is now pinned per-worker rather than following ELAN_HOME under the toolchain directory: it is written throughout a build, not once at install, and that also keeps a one-worker LAKE_ARTIFACT_CACHE experiment honest. The migration script verifies before it deletes. Equal name plus equal size was not the proof the first draft claimed: a `.ltar` is named by a hash of build inputs rather than of its bytes, so pruning now requires the same inode, the same symlink target, or an actual byte comparison, deletes the verified paths one by one instead of removing a tree that also holds files it never examined, and leaves Lake's mutable per-toolchain store out of the pool entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rt6ZAuD5HvTVj1GVexU6CG --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
marcelolynch
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR names the cache tool's temporary files after the writing process, so two
cacheruns sharing one cache directory cannot write each other's curl configuration or each other's partial downloads.The cache directory is per user rather than per checkout (
~/.cache/mathlib), which is what lets a machine pool its downloads. Several runs can therefore be in flight in it at once — different projects, several worktrees of one project, or several agents on one host — and both temporary paths inside it were fixed names.curl.cfgis written, passed to curl, and deleted. A second run overwriting it in the window before curl reads it makes the first run fetch the second's file list; the first run then reports the files it was actually asked for as missing and rebuilds them, which is a full Mathlib compile in the bad case. Whichever run finishes second also throws an uncaughtremoveFileon a path the first already deleted.<hash>.ltar.partis the more damaging one: two curls writing one path interleave their bytes, and whichever rename lands leaves a corrupt.ltarin the shared cache that every later run finds, trusts, and fails to decompress, until someone deletes it by hand.Both now carry the process id, so a run only renames or removes files it wrote, and two runs fetching the same hash simply both produce a complete file and the second rename wins. Renaming and remove-on-error key off this process's own suffix rather than a bare
.partfor the same reason.hashFromFileNamestill reads an untagged<hash>.ltar.part, since a cache directory written by an earlier version may hold some, andlake exe cache-testcovers all three name shapes plus the properties the download monitor relies on.Verified against the real cache server: a single
cache getdownloads, renames, and leaves no temporaries behind, and two concurrentcache get-runs of the same 70-file target into one fresh cache directory both exit 0, leave 70 valid.ltarfiles and no.partor config leftovers, and decompress cleanly afterwards. The failure this fixes is timing-dependent, so the argument for it is the shared fixed paths rather than a deterministic reproduction.🤖 Prepared with Claude Code