[Benchmark] Add all reduce benchmark

[joydddd]

Stacked PRs:

• [Benchmark] Add all gather matmul benchmark #400
• ->[Benchmark] Add all reduce benchmark #393

---

[Benchmark] Add all reduce benchmark

[joydddd]

shape	dtype	multimem	oneshot	twoshot	nccl	helion_oneshot	kraken_oneshot	Best Backend
(4093)	torch.bfloat16	nan	nan	nan	32.812	17.448	18.285	helion_oneshot
(4096)	torch.bfloat16	12.399	16.412	19.269	22.764	14.873	14.254	multimem
(5000)	torch.bfloat16	15.321	16.834	15.442	27.810	15.044	13.743	kraken_oneshot
(8192)	torch.bfloat16	11.815	17.330	15.885	22.541	15.118	13.736	multimem
(8193)	torch.bfloat16	nan	nan	nan	26.715	19.025	19.417	helion_oneshot
(16k)	torch.bfloat16	12.031	17.499	15.525	24.240	15.658	13.852	multimem
(16380)	torch.bfloat16	14.134	16.840	15.759	29.297	15.565	14.471	multimem
(16387)	torch.bfloat16	nan	nan	nan	34.670	22.661	19.729	kraken_oneshot
(32k)	torch.bfloat16	12.058	18.290	16.244	24.535	16.866	15.408	multimem
(64k)	torch.bfloat16	12.663	20.521	18.591	24.497	19.169	17.372	multimem
(128k)	torch.bfloat16	12.912	25.540	19.263	24.780	24.817	21.413	multimem
(256k)	torch.bfloat16	13.796	30.951	18.982	25.564	33.128	29.261	multimem
(512k)	torch.bfloat16	16.783	43.295	19.861	33.252	55.684	40.593	multimem
(1m)	torch.bfloat16	20.311	66.030	23.859	55.766	103.020	62.850	multimem

Helion overhead compared to Kraken (Triton baseline)

1. Helion launches an additional kernel to convert pointer to signal_pad_ptrs array to Tensor.

Kernel+Launch overhead: 2-3us

2. Additional for loop to create multicast_tile: can be solved by enabling [:] indexing with hl.wait / hl.signal
   performance overhead is minor

3. for (512k) & (1m): optimal config uses persistent kernel with partial SMs + pre-log.
   Kraken config: NUM_SM = 24. Each SM conduct symmetric_memory sync outside of the virtual_pid for loop.

4. Helion unrolls tuple of tensor with the same shape, stride & offset, and calculates the offset for each tensor separately.

[joydddd]

Use custom cpp from_blob mod to convert signal_pad_ptrs_dev to tensor.

Benchmarking results for allreduce on 8x devices. (time_us)

shape	dtype	multimem	oneshot	twoshot	nccl	helion_oneshot	kraken_oneshot	Best Backend
(4093)	torch.bfloat16	nan	nan	nan	32.835	15.110	17.817	helion_oneshot
(4096)	torch.bfloat16	12.070	16.807	15.716	28.733	12.797	13.692	multimem
(5000)	torch.bfloat16	11.606	16.931	16.006	27.992	12.456	13.839	multimem
(8192)	torch.bfloat16	11.983	16.828	15.531	22.860	12.526	14.152	multimem
(8193)	torch.bfloat16	nan	nan	nan	26.676	16.445	19.056	helion_oneshot
(16k)	torch.bfloat16	11.875	19.216	18.553	24.183	12.978	14.349	multimem
(16380)	torch.bfloat16	12.047	17.193	15.615	29.370	12.844	14.334	multimem
(16387)	torch.bfloat16	nan	nan	nan	29.815	17.536	20.474	helion_oneshot
(32k)	torch.bfloat16	12.202	17.960	16.332	24.358	15.665	14.748	multimem
(64k)	torch.bfloat16	12.552	20.044	18.470	24.726	16.546	17.160	multimem
(128k)	torch.bfloat16	12.862	25.099	18.502	24.946	20.598	20.648	multimem
(256k)	torch.bfloat16	14.038	30.213	19.259	25.010	30.090	29.323	multimem
(512k)	torch.bfloat16	17.003	43.470	19.799	33.173	50.720	40.312	multimem
(1m)	torch.bfloat16	20.393	65.007	23.570	57.914	91.041	63.734	multimem

Now our performance gap between Helion & Kraken only exist for shape >= 512k where optimal config uses persistent kernel with partial SMs + pre-log.
Kraken config: NUM_SM = 24. Each SM conduct symmetric_memory sync outside of the virtual_pid for loop.

[joydddd]

@yf225 Will Leaving this distributed benchmark PR to you~