What happens
ctest -L gh fails on any build configured with GGML_BACKEND_DL=ON. Two of
the four gh-labelled tests abort:
2/4 Test #11: test-whisper-buffer-loader ....... Passed 0.00 sec
3/4 Test #12: test-whisper-zero-samples ........Subprocess aborted***Exception
4/4 Test #15: test-parakeet ....................Subprocess aborted***Exception
50% tests passed, 2 tests failed out of 4
Both abort the same way:
whisper_init_with_params_no_state: devices = 0
whisper_init_with_params_no_state: backends = 0
ggml/src/ggml-backend.cpp:595: GGML_ASSERT(device) failed
The same test binaries pass when the build does not set GGML_BACKEND_DL.
Why
With GGML_BACKEND_DL=ON the backends are loadable modules, and nothing is
registered until something calls ggml_backend_load_all(). The examples do
this — examples/cli/cli.cpp, examples/bench/bench.cpp,
examples/quantize/quantize.cpp, examples/lsp/lsp.cpp,
examples/vad-speech-segments/speech.cpp. The test programs never do, so
devices stays 0, ggml_backend_dev_backend_reg() gets a null device, and the
assert fires.
That also explains the split in the results:
| test |
outcome |
why |
test-whisper-cli-tiny |
passes |
drives whisper-cli, which calls ggml_backend_load_all() |
test-whisper-buffer-loader |
passes |
never needs a device |
test-whisper-zero-samples |
aborts |
no backend registered |
test-parakeet |
aborts |
no backend registered |
Why CI does not catch it
No workflow currently pairs the two. The workflows that run ctest -L gh —
build-clang.yml, build-gcc.yml, build-sanitize.yml — never set
GGML_BACKEND_DL. The workflow that does set it, release.yml, runs no
ctest at all.
Reproducing
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF
cmake --build build --config Release -j
ctest --test-dir build -L gh --output-on-failure
Seen on macOS 26.6.2 / arm64 with Apple clang 21.0.0, on both a
GGML_CPU_ALL_VARIANTS=ON and a plain GGML_BACKEND_DL=ON build. Nothing in
it looks platform-specific.
Fix
Adding ggml_backend_load_all() to the affected tests, the way the examples
do, should be all it takes. Happy to send that as a PR if the approach looks
right — or if the preference is for the tests to stay backend-agnostic and for
CI to cover the combination instead, that works too.
Noticed while adding the macOS release jobs in #4029, which builds with
GGML_BACKEND_DL=ON; that PR deliberately adds no ctest step because of this.
What happens
ctest -L ghfails on any build configured withGGML_BACKEND_DL=ON. Two ofthe four
gh-labelled tests abort:Both abort the same way:
The same test binaries pass when the build does not set
GGML_BACKEND_DL.Why
With
GGML_BACKEND_DL=ONthe backends are loadable modules, and nothing isregistered until something calls
ggml_backend_load_all(). The examples dothis —
examples/cli/cli.cpp,examples/bench/bench.cpp,examples/quantize/quantize.cpp,examples/lsp/lsp.cpp,examples/vad-speech-segments/speech.cpp. The test programs never do, sodevicesstays 0,ggml_backend_dev_backend_reg()gets a null device, and theassert fires.
That also explains the split in the results:
test-whisper-cli-tinywhisper-cli, which callsggml_backend_load_all()test-whisper-buffer-loadertest-whisper-zero-samplestest-parakeetWhy CI does not catch it
No workflow currently pairs the two. The workflows that run
ctest -L gh—build-clang.yml,build-gcc.yml,build-sanitize.yml— never setGGML_BACKEND_DL. The workflow that does set it,release.yml, runs noctestat all.Reproducing
Seen on macOS 26.6.2 / arm64 with Apple clang 21.0.0, on both a
GGML_CPU_ALL_VARIANTS=ONand a plainGGML_BACKEND_DL=ONbuild. Nothing init looks platform-specific.
Fix
Adding
ggml_backend_load_all()to the affected tests, the way the examplesdo, should be all it takes. Happy to send that as a PR if the approach looks
right — or if the preference is for the tests to stay backend-agnostic and for
CI to cover the combination instead, that works too.
Noticed while adding the macOS release jobs in #4029, which builds with
GGML_BACKEND_DL=ON; that PR deliberately adds nocteststep because of this.