fix: share zig's global cache and retry its LLVM registry race

[basvandijk]

Stacked on #10981 so bazel-test-all-rbe actually runs on this change.

Why RBE was failing

bazel-test-all-rbe failed on 13355d0 (run) with

error: sub-compilation of compiler_rt failed
  note: LLVM failed to parse 'x86_64-unknown-linux5.10.0-gnu2.31.0': No available targets are compatible with triple "x86_64-unknown-linux5.10.0-gnu2.31.0"
  note: failed to link with LLD: UnableToWriteArchive

and was disabled again in c7ae711. The triple is a red herring: 5.10.0 is zig 0.15.2's default minimum Linux kernel, and it emits that triple on every host (zig build-obj --show-builtin confirms it). What fails is LLVM's target lookup.

A cold zig link fans out ~30 sub-compilations — compiler_rt, ubsan_rt, glibc's crt files and .so stubs, libc++ — concurrently on zig's thread pool. Each registers LLVM targets from its own thread: initializeLLVMTarget() in src/codegen/llvm.zig:982 for the Zig pieces, llvm::InitializeAllTargets() in src/zig_clang_cc1_main.cpp:225 for the C ones. LLVM documents TargetRegistry as not thread-safe — registration must not happen while another thread reads the registry — so a lookup can miss the target that was just registered. It happens roughly once in a few hundred cold links, and both call sites are still unguarded on zig master, so a zig bump would not fix it.

Non-RBE CI barely noticed: it does about one cold link per job. RBE did, because our zig cache was sharded per executor slot, so an invocation started cold in up to 240 shards — about 240 chances per build.

What this changes

The expensive sub-compilations live in zig's global cache (measured: 116 entries there vs 1 in the local cache), which is content addressed and records no paths — a cache warmed by one action serves another at a different execroot. So the global cache stays shared and only the local cache is sharded.

To stop many actions from building those entries simultaneously — concurrent cold writers being what corrupts the cache (uber/hermetic_cc_toolchain#224) and what crashed zig's linker child with SIGABRT — the first link for a given tool/flags/target/toolchain takes an exclusive flock, warms the cache and leaves a marker; everyone else finds the marker and runs concurrently. Compilations sit it out: they build nothing in the global cache, and letting one record a marker would leave the first links to run cold together.

The registry race is internal to a single zig process, so process-level locking cannot fix it. Pinning the warm-up to one CPU would (zig sizes its thread pool from the affinity mask) but costs too much — a cold C++ link is ~67s pinned vs ~8s unpinned, because the libc++ build parallelizes ~7×, and arriving actions wait that out. Instead the wrapper captures zig's stderr and re-runs the command once when a failure carries the race's message. Only the last attempt's stderr is forwarded, so a link that survived on retry stays quiet: rustc's linker_messages lint reports anything the linker printed and would turn the survived flake into a failure under -D warnings (the same trap hermetic_cc_toolchain_lto.patch works around).

No prewarming, no worker-image plumbing, and a zig upgrade invalidates the marker through a stat fingerprint of the zig binary.

Verification

Against the wrapper Bazel actually builds:

scenario	result
8 concurrent links, cold shared cache	8.3s wall; one process warms (62s CPU), seven block then run warm (0.03s CPU each); 1 marker, 116 entries
16 concurrent links, warm	0.12s total
compile-only, cold cache	0 markers, 0 global entries
second flag combination on a warm cache	new marker, no cold work, 0.19s for 8 concurrent
bazel build from a cold cache	succeeds; 1 marker, 117 entries, 1 shard

Regression checks on the stderr change (inherit → capture+forward): genuine compile errors and link errors are still reported with the right exit code, warnings on successful compiles are still forwarded, and -E still streams to stdout.

7 wrapper unit tests pass under the pinned zig 0.15.2, including a real-process test driving a fake tool that fails with the race message once then succeeds, plus a 128 KB stderr case covering the pipe draining. The patch stack applies cleanly onto a fresh v4.3.0 tarball in MODULE.bazel order.

Follow-up

Report the unguarded LLVM target registration upstream — the retry is a workaround.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR updates the hermetic Zig C/C++ toolchain wrapper patch to reduce flakiness and overhead in RBE by (1) sharing Zig’s global cache across executor slots while still sharding the local cache, and (2) adding a warm-up lock/marker plus a targeted retry for a known transient LLVM target-registry race.

Changes:

• Keep ZIG_GLOBAL_CACHE_DIR shared while sharding only ZIG_LOCAL_CACHE_DIR by config/slot-derived suffix.
• Serialize the first cold link per tool/flags/target/toolchain via an flock + marker file to avoid concurrent global-cache writers.
• Capture Zig stderr to detect the transient LLVM registry race and retry once, forwarding only the final attempt’s diagnostics; adds unit tests for the new behavior.

Suppressed comments (2)

bazel/hermetic_cc_toolchain_cache_dir.patch:442

• This test uses a GeneralPurposeAllocator without calling deinit(), which can hide leaks from the ArrayListUnmanaged allocations below. Consider deinitializing the GPA in a defer to avoid silent leaks.

+    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
+    const allocator = gpa.allocator();

bazel/hermetic_cc_toolchain_cache_dir.patch:519

• This test also creates a GeneralPurposeAllocator but never calls deinit(). Adding a defer _ = gpa.deinit(); makes allocator lifetime explicit and prevents silent leaks.

+    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
+    const allocator = gpa.allocator();

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.