Skip to content

tests : ctest -L gh fails on a GGML_BACKEND_DL=ON build #4030

Description

@apollo-2006

What happens

ctest -L gh fails on any build configured with GGML_BACKEND_DL=ON. Two of
the four gh-labelled tests abort:

2/4 Test #11: test-whisper-buffer-loader .......   Passed    0.00 sec
3/4 Test #12: test-whisper-zero-samples ........Subprocess aborted***Exception
4/4 Test #15: test-parakeet ....................Subprocess aborted***Exception

50% tests passed, 2 tests failed out of 4

Both abort the same way:

whisper_init_with_params_no_state: devices    = 0
whisper_init_with_params_no_state: backends   = 0
ggml/src/ggml-backend.cpp:595: GGML_ASSERT(device) failed

The same test binaries pass when the build does not set GGML_BACKEND_DL.

Why

With GGML_BACKEND_DL=ON the backends are loadable modules, and nothing is
registered until something calls ggml_backend_load_all(). The examples do
this — examples/cli/cli.cpp, examples/bench/bench.cpp,
examples/quantize/quantize.cpp, examples/lsp/lsp.cpp,
examples/vad-speech-segments/speech.cpp. The test programs never do, so
devices stays 0, ggml_backend_dev_backend_reg() gets a null device, and the
assert fires.

That also explains the split in the results:

test outcome why
test-whisper-cli-tiny passes drives whisper-cli, which calls ggml_backend_load_all()
test-whisper-buffer-loader passes never needs a device
test-whisper-zero-samples aborts no backend registered
test-parakeet aborts no backend registered

Why CI does not catch it

No workflow currently pairs the two. The workflows that run ctest -L gh
build-clang.yml, build-gcc.yml, build-sanitize.yml — never set
GGML_BACKEND_DL. The workflow that does set it, release.yml, runs no
ctest at all.

Reproducing

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF
cmake --build build --config Release -j
ctest --test-dir build -L gh --output-on-failure

Seen on macOS 26.6.2 / arm64 with Apple clang 21.0.0, on both a
GGML_CPU_ALL_VARIANTS=ON and a plain GGML_BACKEND_DL=ON build. Nothing in
it looks platform-specific.

Fix

Adding ggml_backend_load_all() to the affected tests, the way the examples
do, should be all it takes. Happy to send that as a PR if the approach looks
right — or if the preference is for the tests to stay backend-agnostic and for
CI to cover the combination instead, that works too.

Noticed while adding the macOS release jobs in #4029, which builds with
GGML_BACKEND_DL=ON; that PR deliberately adds no ctest step because of this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions