perf: rewrite square_redc to unroll and use single-word carries

[prestwich]

Summary

square_redc was slower than mul_redc(a, a) at every width on every architecture tested — 2.05× at 256 bits on Apple M-series, ~1.4× on AMD Zen 2 — when squaring should cost at most a multiply. This PR rewrites it so that it wins (or ties) everywhere, and adds the real-world benchmark suite that found and verifies the problem.

Root cause

The old implementation did fewer multiplies on paper but compiled to much worse code, for two structural reasons:

1. The doubled-cross-term inner loop (for j in (i + 1)..N) had a data-dependent, triangular trip count, which LLVM refuses to fully unroll. At N = 4, mul_redc compiles to ~400 straight-line branch-free instructions, while the old square_redc compiled to a ~105-instruction rolled nested loop — with the loop-invariant modulus[N - 1] test re-executed on every outer iteration.
2. 2·a[i]·a[j] + limb + carry overflows 128 bits, forcing a 65-bit (u64, bool) carry. A two-word carry cannot live in the flags register, so it was materialized and re-added on every iteration, serializing the whole cross-term chain.

New algorithm

• Accumulate the undoubled upper-triangle cross products with plain u64 carries; the triangular structure is expressed as guards inside constant-trip loops so everything unrolls and the guards fold away.
• Double the whole 2N-limb product and add the diagonal squares in a single flag-friendly pass.
• Montgomery-reduce with a CIOS-style sliding window (constant indices only), handing the top-limb carry forward across rounds as a single bool.
• For N > 8 the loops no longer unroll and the specialized path loses its edge, so it falls back to mul_redc(a, a) — which measures up to 39% faster there (2048 bits, M-series).

Benchmarks added

Four new bench groups under ruint-bench capture real-world workloads that the existing random-modulus benches miss, and serve as the regression harness for this fix:

• fields/{field}/{op} — fixed-modulus kernels over the 11 moduli that dominate real usage (secp256k1 p/n, P-256 p/n, BN254 p/r, BLS12-381 p/r, BLS12-377 p/r, CSIDH-512), including mul_redc, square_redc, and real fixed exponents (Fermat inversion, sqrt decompression, 65537). Montgomery contexts are self-checked at registration time.
• chained/{op}/{tag} — 64-deep serial dependency chains measuring critical-path latency, which throughput benches and instruction counting are both blind to. Includes const_eq/const_is_zero variants at 256 and 512 bits — the only regression guard for perf: unify const_eq/const_is_zero with their runtime equivalents #615's latency-shaped fold fix (cherry-picked here from perf: scalar-friendly const_eq fold for x86-64 dependent-use latency #617 after it was folded into perf: unify const_eq/const_is_zero with their runtime equivalents #615).
• curves/{curve}/{kernel} — composite kernels (Jacobian double/add with known-answer checks against 2G/3G, Fp2 mul, 16-bit scalar-mul segment, ECDSA verify prologue, 64-element batch inversion).
• dist/{op}/{distribution}/256 — comparisons under realistic input distributions (equal/differ-late, 160-bit addresses, small counters) where branchy vs branchless implementations rank differently. Includes const_eq variants guarding perf: unify const_eq/const_is_zero with their runtime equivalents #615's standalone-throughput tradeoff.

Measurements

Wall clock via fields/*/square_redc (was → now, mul_redc for reference):

width	Apple M-series	AMD Zen 2
256-bit	25.2 → 12.3 ns (mul: 14.5)	32.7 → 23.5 ns (mul: 23.1)
384-bit	49.4 → 21.6 ns (mul: 28.2)	~61 → 42.0 ns (mul: 46.7)
512-bit	72.1 → 37.9 ns (mul: 47.6)	~110 → 79.5 ns (mul: 82.1)

Dependent-chain latency (chained/square_redc/*) improves similarly (e.g. 384-bit on M-series: 1.56 µs vs mul's 2.07 µs per 64-op chain).

Work counts: exact dynamic instruction counts (callgrind, x86-64) are strictly below both the old code and mul_redc(a, a) at 4/6/8 limbs (e.g. 273 vs 327 vs 422 instructions per op at 256 bits), so CodSpeed's instrumentation metric improves as well. Crossover sweep (synthetic moduli at 5, 7, 16, 32 limbs) confirms the N ≤ 8 specialization threshold on both architectures.

Correctness: existing proptest suites (algorithms::mul_redc::test_square_redc, modular::tests::test_square_redc) cover 16–4096 bits against the division-based reference, now extended to 320/448 bits (the odd specialized widths N = 5, 7) and 576 bits (the first fallback width). A directed carry-saturation test squares a = m - 1 under all-ones-shaped moduli at every N ≤ 9, driving the (u64, bool) carry sums to their exact 2^65 - 1 ceiling — the corner uniform-random proptests cannot reach and the classic hiding spot for a lost squaring carry. Green on stable and with the nightly feature. All bench groups run registration-time self-checks (Montgomery round-trips, secp256k1 known-answer points, algebraic identities) so a wrong constant panics instead of benchmarking garbage.

⚠️ Blocker — why this is a draft

Stacked on #615 (prestwich/const-eq-dw): this PR is based on and targets that branch, not main, and is rebased onto its current tip (the XOR/OR-fold const_eq absorbed from #617, plus its cutoff-boundary tests). It must not merge until #615 lands, then be retargeted to main. The square_redc work itself is functionally independent (no file overlap) and could be rebased onto main directly if #615 stalls — but the chained/const_eq and dist/const_eq bench guards in this PR exist to protect #615's fix, so the two should land together.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UJ4S2Ns5cyvmkTFJ7mgrWE

[codspeed-hq]

Merging this PR will not alter performance

✅ 400 untouched benchmarks
🆕 108 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	chained/add_mod/secp256k1_p	N/A	502.7 µs	N/A
🆕	chained/add/256	N/A	23.2 µs	N/A
🆕	chained/const_eq/256	N/A	42.9 µs	N/A
🆕	chained/const_eq/512	N/A	87.9 µs	N/A
🆕	chained/const_is_zero/256	N/A	28.6 µs	N/A
🆕	chained/const_is_zero/512	N/A	50 µs	N/A
🆕	chained/eq/256	N/A	55.7 µs	N/A
🆕	chained/eq/512	N/A	152.6 µs	N/A
🆕	chained/is_zero/256	N/A	28.6 µs	N/A
🆕	chained/is_zero/512	N/A	50 µs	N/A
🆕	chained/mul_redc/bls12_381_p	N/A	1.6 ms	N/A
🆕	chained/mul_redc/secp256k1_p	N/A	740.2 µs	N/A
🆕	chained/mul/256	N/A	132.5 µs	N/A
🆕	chained/square_redc/bls12_381_p	N/A	1.3 ms	N/A
🆕	chained/square_redc/secp256k1_p	N/A	622.4 µs	N/A
🆕	curves/bls12_381_g1/jac_add	N/A	467.9 µs	N/A
🆕	curves/bls12_381_g1/jac_double	N/A	287.7 µs	N/A
🆕	curves/bls12_381/fp2_mul	N/A	120.4 µs	N/A
🆕	curves/secp256k1/batch_inverse64	N/A	2.7 ms	N/A
🆕	curves/secp256k1/ecdsa_verify_prologue	N/A	279.8 µs	N/A
...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---

_{Comparing prestwich/square-redc (e8989d6) with prestwich/const-eq-dw (0421250)}