Windows Vulkan build fails in vulkan-shaders-gen ExternalProject with misleading MSVC C1041 "multiple CL.EXE" PDB error (native, non-cross-compiling build)

[rhybiq]

Title

Windows Vulkan build fails in vulkan-shaders-gen ExternalProject with misleading MSVC C1041 "multiple CL.EXE" PDB error (native, non-cross-compiling build)

Environment

• OS: Windows Server 2025 (GitHub Actions windows-2025 hosted runner)
• Visual Studio: 18 (2026), MSVC 19.51.36252.0 (VCToolsInstallDir ...\VC\Tools\MSVC\14.51.36231\)
• CMake: 4.4 (C:\Program Files\CMake\bin\cmake.exe)
• Ninja: 1.13.2
• Vulkan SDK: 1.4.357.0 (official LunarG installer, silent install)
• ggml version: 0.9.5, ggml commit: cc34d99
• Consumed via whisper-rs-sys v0.15.0 (whisper-rs v0.16.0), built from a Rust/Tauri app via cargo build → cmake crate (not built by invoking cmake/ninja directly)
• Native x64 → x64 build (not cross-compiling)

What happens

Building with the vulkan feature enabled (GGML_VULKAN=ON), the outer whisper.cpp/ggml build configures and starts compiling fine (correctly identifies MSVC 19.51.36252.0, finds Vulkan/glslc/headers/lib without issue). It fails specifically when CMake reaches the vulkan-shaders-gen ExternalProject_Add step (ggml/src/ggml-vulkan/CMakeLists.txt), which runs its own nested cmake -S .../vulkan-shaders -B ... configure. That nested configure's CMakeTestCCompiler.cmake bootstrap try_compile() fails every time with:

-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
CMake Error at .../CMakeTestCCompiler.cmake:65 (message):
  ...
  FAILED: [code=2] CMakeFiles/cmTC_XXXXX.dir/testCCompiler.c.obj
  "...\cl.exe"  /nologo -D_MBCS  /FS  /DWIN32 /D_WINDOWS  /Zi /Ob0 /Od /RTC1 -MDd ... /FoCMakeFiles\cmTC_XXXXX.dir\testCCompiler.c.obj /FdCMakeFiles\cmTC_XXXXX.dir\ /FS -c ...\testCCompiler.c
  ...\testCCompiler.c: fatal error C1041: cannot open program database '...\CMakeFiles\cmTC_XXXXX.dir\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS
  ninja: build stopped: subcommand failed.
-- Configuring incomplete, errors occurred!

Note /FS is already present twice in the actual invocation (once from CMake's own default injection, once from an explicit override I added) — so this is not literally a missing-/FS problem despite the error text. This is a uniquely-hashed scratch directory (CMakeScratch/TryCompile-xxxxxx/CMakeFiles/cmTC_xxxxx.dir/, different every run), so it isn't stale-file contention from a previous run either.

This reproduces 100% of the time, byte-identical error, across many separate CI runs.

What I tried (all failed identically — no change in error text, timing, or behavior)

1. Forcing /FS via CFLAGS/CXXFLAGS env vars.
2. Forcing /FS via the MSVC-native _CL_ env var (which cl.exe itself prepends to every invocation, confirmed present in the logged command line above) — still fails.
3. Serializing the entire build (CMAKE_BUILD_PARALLEL_LEVEL=1) to rule out genuine cross-process races — same failure, same wall-clock timing, suggesting it isn't really a concurrency race.
4. Forcing the nested build onto NMake Makefiles via CMAKE_GENERATOR env var — got further (past compiler identification!) but then hit NMAKE : fatal error U1052: file '...build.make' not found, a known NMake long-path weakness.
5. Switching to CMAKE_GENERATOR=Ninja instead — compiler identification succeeds (MSVC 19.51.36252.0), but the PDB C1041 error above appears instead.
6. Swapping which ninja.exe is used (Chocolatey's vs. the one bundled with the VS install) — both are the identical 1.13.2, no difference.
7. Excluding the whole workspace + cl.exe/ninja.exe from Windows Defender real-time scanning (Add-MpPreference -ExclusionPath/-ExclusionProcess) — no change.
8. Enabling Windows long-path support (HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled=1 + git config --system core.longpaths true) — measured the actual failing PDB path at 264 characters (over the classic 260 MAX_PATH limit), but enabling long-path support made no difference either.
9. Disabling ccache (GGML_CCACHE=OFF, confirmed it took effect — no "ccache found" in the log) — no change.

Suspected root cause

ggml/src/ggml-vulkan/CMakeLists.txt only sets up an explicit host toolchain (HOST_CMAKE_TOOLCHAIN_FILE, via detect_host_compiler() or GGML_VULKAN_SHADERS_GEN_TOOLCHAIN) for the vulkan-shaders-gen ExternalProject_Add when CMAKE_CROSSCOMPILING is true:

if (CMAKE_CROSSCOMPILING)
    if (GGML_VULKAN_SHADERS_GEN_TOOLCHAIN)
        set(HOST_CMAKE_TOOLCHAIN_FILE ${GGML_VULKAN_SHADERS_GEN_TOOLCHAIN})
    else()
        detect_host_compiler()
        ...
    endif()
else()
    # For non-cross-compiling, use empty toolchain (use host compiler)
    set(HOST_CMAKE_TOOLCHAIN_FILE "")
endif()

For a native build (our case), no toolchain/compiler/generator override is passed to the ExternalProject_Add at all — the nested configure is left entirely to CMake's own bare default-generator/compiler discovery. That discovery appears to be unreliable in this environment (possibly specific to the newer VS "18"/2026 toolset not yet being recognized cleanly by this CMake version's default-generator heuristics), and whatever underlying Win32 failure results gets surfaced by cl.exe as the generic (and here, misleading) "multiple CL.EXE" PDB message rather than the actual cause.

Ask

Could the native (non-cross-compiling) branch also pass an explicit, resolved compiler/generator to the vulkan-shaders-gen ExternalProject_Add (e.g. always run detect_host_compiler() / generate a host toolchain file, not just when CMAKE_CROSSCOMPILING), rather than relying on bare default discovery? Happy to test a patch or provide more logs/repro details.

---

Repro repo (not currently public/linked): building whisper-rs-sys 0.15.0 with the vulkan feature via a Tauri app's cargo build, on GitHub Actions windows-2025. Can provide the full workflow YAML and CI logs on request.