perf: scalar-friendly const_eq fold for x86-64 dependent-use latency

[prestwich]

Summary

Reformulates const_eq's ≤16-limb path from u128 double-word chunks to a limb-wise XOR/OR fold, fixing a large dependent-use latency penalty on x86-64 while keeping the aarch64 codegen from #615 bit-identical. Measured on Zen 2 (Threadripper 3990X): −44% per-link latency for const_eq at 256 bits (3.71 → 2.09 ns), −52% at 512 bits, with all distribution and throughput guards flat. On Apple M-series the same change is −18.5% at 256 bits.

Stacks on #615 (its two commits are included here) and on the real-world benchmark suite this investigation used; draft until both land or this gets rebased onto them.

The problem

The chained/* latency benches (64 serially-dependent op applications, so the CPU cannot overlap iterations) showed that consuming an equality result costs far more than producing one on Zen 2:

bench (Zen 2)	standalone	chained, per link
eq/256 (derived ==)	1.30 ns	8.45 ns
const_eq/256 (#615 dw form)	0.85 ns	3.71 ns

Root cause, from the generated asm: both forms lower to an SSE sequence (movdqu×4, pxor×2, por, ptest, sete). In a dependency chain the consumer updates the limbs with scalar code (4×8-byte stores from an adc chain), and the next compare reloads them as 16-byte vector loads. A 16-byte load spanning two 8-byte stores fails store-to-load forwarding and stalls until the stores drain — that, plus the ptest → sete vector→scalar domain crossing, is the 8.45 ns/link. This is the x86-64 sibling of the aarch64 NEON/fmov pathology #615 fixed, but worse (6.5× vs 4×).

The fix

// LIMBS <= EQ_FOLD_MAX_LIMBS (16):
let mut acc = 0;
const_range_for!(i in 0..LIMBS => {
    acc |= a[i] ^ b[i];
});
acc == 0

No cfg(target_arch) needed — LLVM canonicalizes the &-chain of limb equalities, the u128 double-word chain, and this fold to identical IR on aarch64 (all lower to the same ccmp flag chain #615 chose, verified by symbol-aliased asm), but on x86-64 only the u64 fold escapes re-vectorization when the result feeds a dependency chain:

context	dw chunks (#615)	limb-wise fold (this PR)
x86-64, dependent use	pxor/ptest + STLF stall	pure-GPR xor/or, no memory round-trip
x86-64, standalone	pxor/ptest	ptest at 8+ limbs; scalar at 4 limbs
aarch64, dependent use	ccmp flag chain	ccmp flag chain (identical)
aarch64, standalone	ccmp	NEON cmeq/umaxv

is_zero/const_is_zero inherit the fix through the existing const_eq(&ZERO) routing; the > 16 limb paths (counting loop, OR-fold, bcmp against ZERO) are untouched.

Measurements

Criterion, chained/* reported per link (reported time ÷ 64). Zen 2 = Threadripper 3990X pinned with taskset; M-series = Apple Silicon.

Latency (the target):

bench	Zen 2 before → after	M-series before → after
chained/const_eq/256	3.71 → 2.09 ns (−44%)	2.26 → 1.83 ns (−18.5%)
chained/const_eq/512	~6.5 (unstable) → 3.40 ns (−52%)	3.33 → 2.96 ns (−11%)
chained/is_zero/256	1.75 → 1.72 ns (−3.5%)	1.87 → 1.88 ns (flat)
chained/const_is_zero/256	1.73 → 1.72 ns	1.83 → 1.89 ns
is_zero/384, /512 standalone	−18% … −23%	−3.5%

Guards (must not move, and didn't):

bench	Zen 2	M-series
dist/eq/{equal,differ_low,differ_high,mixed50}/256	−2%	flat
dist/is_zero/mixed10/256	flat	flat
chained/eq/* (derived ==, untouched)	+2% (noise)	flat
eq/is_zero/const_* at 64–192, 4096 bits	noise band (±7%)	noise band

Known tradeoff: standalone const_eq/256 throughput on Zen 2 goes 0.85 → 1.12 ns (+34%; dist/const_eq/* show the same, flat across distributions since both forms are branchless). At exactly 4 limbs x86-64 keeps even the standalone fold scalar; at 8+ limbs it re-vectorizes (const_eq/512 standalone is unchanged). 1.12 ns still beats the derived == (1.32 ns) on the same inputs, and an equality result in real code is almost always consumed dependently — which is where the 44% win is — so this looks like the right trade. Documented in the EQ_FOLD_MAX_LIMBS comment.

What this can't fix

chained/eq/256 — the derived == itself — stays at ~8.5 ns/link on Zen 2. #[derive(PartialEq)] must remain (removing it breaks StructuralPartialEq, i.e. Uint constants in match patterns), so its bcmp-shaped lowering is not ours to change. Incidentally, the benches show how fragile that lowering is: the untouched eq/192 flipped from 1.5 ns (inlined) to 7.8 ns (out-of-line bcmp call) purely from unrelated code motion in the bench binary, and eq/384/eq/512 already sit at ~10.5 ns standalone on Linux/Zen 2. Latency-sensitive code should prefer const_eq/is_zero.

Commits

• 7 × bench: — the real-world benchmark suite (fields/chained/curves/dist groups) this investigation was run with
• 2 × perf: … (cherry-picks of perf: unify const_eq/const_is_zero with their runtime equivalents #615)
• bench: add const_eq/const_is_zero chained + dist variants, 512-bit widths — new guards used above
• perf: reformulate const_eq fold to stay scalar under dependent use — the fix

Testing

• cargo test --lib: 147 passed (includes test_const_eq_ctfe compile-time evaluation, proptest equivalence of const_eq/const_is_zero vs ==/is_zero across all SIZES, and single-limb-difference detection)
• cargo clippy --all-targets: clean; cargo +nightly fmt --check: clean
• Asm inspected on both targets (x86-64 generic and aarch64) for standalone and chained contexts at 4, 8, and 16 limbs

🤖 Generated with Claude Code

[prestwich]

Folded into #615: the const_eq XOR/OR-fold reformulation was cherry-picked onto prestwich/const-eq-dw (47de7fd), and #615's description now carries the x86-64 STLF analysis, measurements, and known tradeoffs from this PR. Note: the bench-guard commit here (6b7bd7d, chained/const_eq/dist/const_eq variants) depends on the real-world bench suite files and is not part of #615 — it should be cherry-picked into #616 (or wherever the bench suite lands) so the wall-clock latency guards survive.