Repository navigation
Make the performance tests robust to timer granularity - #30
Merged
yohanchatelain merged 1 commit intoOct 8, 2026
Merged
Conversation
The benchmarks timed every call individually and reported mean ± std over 100k calls. On systems with a coarse timer (~32 ns on some Arm cores, about the cost of one vector operation) single-vector timings were mostly quantization noise (e.g. 54 ns ± 720 ns, min 0), and the mean was dominated by interruptions. Add tests/helper/benchmark.h, shared by the SR and UD performance tests: calls are batched so that each timed sample lasts at least 50 us, and the median and 10th/90th percentiles over 101 samples are reported, per call and per element. Medians reproduce within ~0.7% between runs, and each performance test now runs in about a second instead of 15-30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 5 of the experimental Arm support: benchmarks that give stable numbers on Arm, plus a first NEON vs SVE comparison.
Problem
The performance tests timed each call on its own and averaged over a fixed number of repetitions. On Arm the system timer ticks every ~32 ns, about the cost of a single vector operation (
addf32x4_v≈ 18 ns), so per-call timings were mostly timer noise. Each test also took 15–30 s.Change
tests/helper/benchmark.h. It times batches of calls withsteady_clock, doubling the batch size until a batch takes at least 50 µs, then takes 101 samples. It reports the median and the 10th/90th percentiles per call and per element.test_sr_performance.cppandtest_ud_performance.cppuse the new helper. The fixed repetition count and the manual timing loops are removed.Repeated runs reproduce the medians within 0.7% on average (4.3% at worst), and each test now takes about 1 s.
NEON vs SVE on GB10 (Cortex-X925, 128-bit SVE2)
Each value is SVE time divided by NEON time (geometric mean), so above 1 means SVE is slower. The tests were pinned to one core and built with
-march=armv9.2-a+sve2, using+nosvefor the NEON build.On this core SVE is never faster for arrays: about 13% slower for SR and 50–67% slower for UD. The one SVE win is clang 22 on SR static single vectors (12% faster). The static UD gap reflects the target choice: the static library is built for scalable SVE2, while dynamic dispatch selects the fixed 128-bit SVE2_128 target. These results support keeping SVE opt-in (#28). With NEON, clang 22 is 2–9% faster than GCC on arrays.
I couldn't profile on this machine (
perf_event_paranoid=4), so the cause of the SVE slowdown is still open. The clang 22 numbers were taken while another workload ran on other cores, so expect a few percent of noise.🤖 Generated with Claude Code