Skip to content

Make the performance tests robust to timer granularity - #30

Merged
yohanchatelain merged 1 commit into
verificarlo:mainfrom
yohanchatelain:bench-batch-timing
Oct 8, 2026
Merged

yohanchatelain merged 1 commit into
verificarlo:mainfrom
yohanchatelain:bench-batch-timing

Conversation

@yohanchatelain

Copy link
Copy Markdown
Collaborator

Phase 5 of the experimental Arm support: benchmarks that give stable numbers on Arm, plus a first NEON vs SVE comparison.

Problem

The performance tests timed each call on its own and averaged over a fixed number of repetitions. On Arm the system timer ticks every ~32 ns, about the cost of a single vector operation (addf32x4_v ≈ 18 ns), so per-call timings were mostly timer noise. Each test also took 15–30 s.

Change

  • New tests/helper/benchmark.h. It times batches of calls with steady_clock, doubling the batch size until a batch takes at least 50 µs, then takes 101 samples. It reports the median and the 10th/90th percentiles per call and per element.
  • test_sr_performance.cpp and test_ud_performance.cpp use the new helper. The fixed repetition count and the manual timing loops are removed.

Repeated runs reproduce the medians within 0.7% on average (4.3% at worst), and each test now takes about 1 s.

NEON vs SVE on GB10 (Cortex-X925, 128-bit SVE2)

Each value is SVE time divided by NEON time (geometric mean), so above 1 means SVE is slower. The tests were pinned to one core and built with -march=armv9.2-a+sve2, using +nosve for the NEON build.

Suite clang 22, arrays of 1024 GCC, arrays of 1024 clang 22, single 128-bit vector GCC, single 128-bit vector
SR static 1.13× 1.13× 0.88× 1.08×
SR dynamic 1.13× 1.14× 0.98× 0.99×
UD static 1.64× 1.52× 1.69× 1.45×
UD dynamic 1.67× 1.54× 1.00× 0.97×

On this core SVE is never faster for arrays: about 13% slower for SR and 50–67% slower for UD. The one SVE win is clang 22 on SR static single vectors (12% faster). The static UD gap reflects the target choice: the static library is built for scalable SVE2, while dynamic dispatch selects the fixed 128-bit SVE2_128 target. These results support keeping SVE opt-in (#28). With NEON, clang 22 is 2–9% faster than GCC on arrays.

I couldn't profile on this machine (perf_event_paranoid=4), so the cause of the SVE slowdown is still open. The clang 22 numbers were taken while another workload ran on other cores, so expect a few percent of noise.

🤖 Generated with Claude Code

The benchmarks timed every call individually and reported mean ± std over
100k calls. On systems with a coarse timer (~32 ns on some Arm cores, about
the cost of one vector operation) single-vector timings were mostly
quantization noise (e.g. 54 ns ± 720 ns, min 0), and the mean was dominated
by interruptions.

Add tests/helper/benchmark.h, shared by the SR and UD performance tests:
calls are batched so that each timed sample lasts at least 50 us, and the
median and 10th/90th percentiles over 101 samples are reported, per call and
per element. Medians reproduce within ~0.7% between runs, and each
performance test now runs in about a second instead of 15-30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yohanchatelain
yohanchatelain merged commit 79fab6a into verificarlo:main Oct 8, 2026
14 of 15 checks passed
@yohanchatelain
yohanchatelain deleted the bench-batch-timing branch October 8, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant