Skip to content

fix(Cache): give each cache process its own temporary file names - #42752

Open
kim-em wants to merge 1 commit into
leanprover-community:masterfrom
kim-em:cache-concurrent-dir
Open

fix(Cache): give each cache process its own temporary file names#42752
kim-em wants to merge 1 commit into
leanprover-community:masterfrom
kim-em:cache-concurrent-dir

Conversation

@kim-em

@kim-em kim-em commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

This PR names the cache tool's temporary files after the writing process, so two cache runs sharing one cache directory cannot write each other's curl configuration or each other's partial downloads.

The cache directory is per user rather than per checkout (~/.cache/mathlib), which is what lets a machine pool its downloads. Several runs can therefore be in flight in it at once — different projects, several worktrees of one project, or several agents on one host — and both temporary paths inside it were fixed names.

curl.cfg is written, passed to curl, and deleted. A second run overwriting it in the window before curl reads it makes the first run fetch the second's file list; the first run then reports the files it was actually asked for as missing and rebuilds them, which is a full Mathlib compile in the bad case. Whichever run finishes second also throws an uncaught removeFile on a path the first already deleted. <hash>.ltar.part is the more damaging one: two curls writing one path interleave their bytes, and whichever rename lands leaves a corrupt .ltar in the shared cache that every later run finds, trusts, and fails to decompress, until someone deletes it by hand.

Both now carry the process id, so a run only renames or removes files it wrote, and two runs fetching the same hash simply both produce a complete file and the second rename wins. Renaming and remove-on-error key off this process's own suffix rather than a bare .part for the same reason. hashFromFileName still reads an untagged <hash>.ltar.part, since a cache directory written by an earlier version may hold some, and lake exe cache-test covers all three name shapes plus the properties the download monitor relies on.

Verified against the real cache server: a single cache get downloads, renames, and leaves no temporaries behind, and two concurrent cache get- runs of the same 70-file target into one fresh cache directory both exit 0, leave 70 valid .ltar files and no .part or config leftovers, and decompress cleanly afterwards. The failure this fixes is timing-dependent, so the argument for it is the shared fixed paths rather than a deterministic reproduction.

🤖 Prepared with Claude Code

The cache directory is per user, not per checkout, so several `cache` runs
share it: different projects, several worktrees of one project, or a fleet of
agents on one machine. Both temporary paths inside it were fixed names, so
those runs wrote each other's files.

`curl.cfg` is written, handed to curl, and deleted. When a second run
overwrote it in the window before curl read it, the first run downloaded the
second's file list, then reported its own files as missing and rebuilt them;
whichever run finished second also hit an uncaught exception deleting a file
that was no longer there. `<hash>.ltar.part` is worse: two curls writing one
path interleave, and the winner of the rename leaves a corrupt `.ltar` that
every later run finds, trusts, and fails to decompress.

Name both after the writing process, so a run only ever renames or removes
its own files. `hashFromFileName` keeps reading untagged `.ltar.part` names,
which a cache directory written by an older version may still hold.
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

PR summary 50b5b91641

Import changes for modified files

No significant changes to the import graph

Import changes for all files
Files Import difference

Declarations diff (regex)

+ PARTSUFFIX
+ test_tempFileNames

You can run this locally as follows
## from your `mathlib4` directory:
git clone https://github.com/leanprover-community/mathlib-ci.git ../mathlib-ci

## summary with just the declaration names:
../mathlib-ci/scripts/pr_summary/declarations_diff.sh <optional_commit>

## more verbose report:
../mathlib-ci/scripts/pr_summary/declarations_diff.sh long <optional_commit>

The doc-module for scripts/pr_summary/declarations_diff.sh in the mathlib-ci repository contains some details about this script.

Declarations diff (Lean)

Lean-aware diff — post-build, computed from the Lean environment (commit 50b5b91).

  • +0 new declarations
  • −0 removed declarations

No declaration differences.


No changes to strong technical debt.

No changes to weak technical debt.

Current commit 50b5b91641
Reference commit caa4dbde35

This script lives in the mathlib-ci repository. To run it locally, from your mathlib4 directory:

git clone https://github.com/leanprover-community/mathlib-ci.git ../mathlib-ci
../mathlib-ci/scripts/reporting/technical-debt-metrics.sh pr_summary
  • The relative value is the weighted sum of the differences with weight given by the inverse of the current value of the statistic.
  • The absolute value is the relative value divided by the total sum of the inverses of the current values (i.e. the weighted average of the differences).

@github-actions github-actions Bot added the CI Modifies the continuous integration setup or other automation label Aug 13, 2026
kim-em added a commit to kim-em/TauCetiWorker that referenced this pull request Aug 14, 2026
From a review of the first draft, which pointed every worker's
MATHLIB_CACHE_DIR at one directory. That reintroduced a race the per-worker
homes had been hiding: `lake exe cache get` takes no lock, and until every
checkout runs a Mathlib carrying leanprover-community/mathlib4#42752, two
concurrent runs share `curl.cfg` and share each `<hash>.ltar.part`. The
outcome is not a failed round but a corrupt `.ltar` under a name every later
run trusts, for every worker and the operator alike — and the risk peaks
exactly when the pool pays off most, just after a bump, when several workers
fetch the same 8,600 files at once.

Each worker now downloads into its own directory and exchanges finished files
with the pool by hardlink before the agent starts: it promotes what it fetched
last round, then takes a link to everything the pool has that it lacks. Only
complete artifacts reach the pool, an existing name is never replaced, and a
link is not a copy, so the download saving stands and the disk saving with it.

Toolchains are still shared outright, since an elan install takes a
per-toolchain lock and lands by rename. LAKE_CACHE_DIR is now pinned
per-worker rather than following ELAN_HOME under the toolchain directory: it
is written throughout a build, not once at install, and that also keeps a
one-worker LAKE_ARTIFACT_CACHE experiment honest.

The migration script verifies before it deletes. Equal name plus equal size
was not the proof the first draft claimed: a `.ltar` is named by a hash of
build inputs rather than of its bytes, so pruning now requires the same inode,
the same symlink target, or an actual byte comparison, deletes the verified
paths one by one instead of removing a tree that also holds files it never
examined, and leaves Lake's mutable per-toolchain store out of the pool
entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rt6ZAuD5HvTVj1GVexU6CG
@bryangingechen bryangingechen self-assigned this Aug 14, 2026
kim-em added a commit to kim-em/TauCetiWorker that referenced this pull request Aug 14, 2026
* fix: pool the Lean build caches across workers

Mathlib's cache tool resolves ~/.cache/mathlib and elan resolves ~/.elan
through $HOME, so the per-worker $HOME that isolate_home() creates for
credentials also gave every worker a private copy of the same public,
content-addressed downloads. On a five-worker fleet that cost half of one
week's 10.2 GB of .ltar traffic to Mathlib's cache server, and 22 toolchain
installs covering 6 distinct toolchains.

Export MATHLIB_CACHE_DIR and ELAN_HOME from both isolate_home() paths,
resolved through _host_home() so the already-isolated path a round child
runs computes the same pool without waiting for a fleet restart. An
operator-set value still wins. Lake's artifact cache lives under the
toolchain directory, so it follows ELAN_HOME; the bubble environment drops
both, since the container is configured by Bubble's own cache proxy.

scripts/share-build-caches folds pre-existing per-worker copies into the
pool by hardlink, and prunes them only where the pool already holds the
same path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rt6ZAuD5HvTVj1GVexU6CG

* style: ruff format the new test and script

* fix: pool the Mathlib cache by hardlink instead of by shared directory

From a review of the first draft, which pointed every worker's
MATHLIB_CACHE_DIR at one directory. That reintroduced a race the per-worker
homes had been hiding: `lake exe cache get` takes no lock, and until every
checkout runs a Mathlib carrying leanprover-community/mathlib4#42752, two
concurrent runs share `curl.cfg` and share each `<hash>.ltar.part`. The
outcome is not a failed round but a corrupt `.ltar` under a name every later
run trusts, for every worker and the operator alike — and the risk peaks
exactly when the pool pays off most, just after a bump, when several workers
fetch the same 8,600 files at once.

Each worker now downloads into its own directory and exchanges finished files
with the pool by hardlink before the agent starts: it promotes what it fetched
last round, then takes a link to everything the pool has that it lacks. Only
complete artifacts reach the pool, an existing name is never replaced, and a
link is not a copy, so the download saving stands and the disk saving with it.

Toolchains are still shared outright, since an elan install takes a
per-toolchain lock and lands by rename. LAKE_CACHE_DIR is now pinned
per-worker rather than following ELAN_HOME under the toolchain directory: it
is written throughout a build, not once at install, and that also keeps a
one-worker LAKE_ARTIFACT_CACHE experiment honest.

The migration script verifies before it deletes. Equal name plus equal size
was not the proof the first draft claimed: a `.ltar` is named by a hash of
build inputs rather than of its bytes, so pruning now requires the same inode,
the same symlink target, or an actual byte comparison, deletes the verified
paths one by one instead of removing a tree that also holds files it never
examined, and leaves Lake's mutable per-toolchain store out of the pool
entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rt6ZAuD5HvTVj1GVexU6CG

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Modifies the continuous integration setup or other automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants