Skip to content

Commit 81191dd

Browse files
Abraham Sewillclaude
andcommitted
readme: document 3-tier streaming dispatch (pool | plain | compact)
Requirements block, env vars table, and VRAM section updated to describe the new plain tier that sits between pool and compact: ~7.3 GB peak, no park/rehydrate round-trips, ~400 ms/plot faster than compact. Perf table gains rows for both streaming tiers with the measured s/plot on an RTX 4090. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 0887dde commit 81191dd

1 file changed

Lines changed: 36 additions & 24 deletions

File tree

‎README.md‎

Lines changed: 36 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,19 @@ GPU plotter for Chia v2 proofs of space (CHIP-48). Produces farmable
3939
from `rocminfo` automatically. Other gfx targets (`gfx1030` /
4040
`gfx1100`) build cleanly but are untested on real hardware.
4141
- **Intel oneAPI** is wired up but untested.
42-
- **VRAM:** ~5.4 GB free minimum for k=28 (streaming path). Cards
43-
with less than ~11 GB free transparently use the streaming pipeline;
44-
12 GB+ cards reliably use the persistent buffer pool for faster
45-
steady-state. Both paths produce byte-identical plots. 6 GB cards
46-
(RTX 2060, RX 6600) are on the edge and 8 GB cards (3070, 2070 Super)
47-
are comfortably supported on the streaming path — peak is 5200 MB.
48-
Detailed breakdown in [VRAM](#vram).
42+
- **VRAM:** three tiers, picked automatically based on free device
43+
VRAM at k=28. All three produce byte-identical plots.
44+
- **Pool** (~11 GB device + ~4 GB pinned host): fastest steady-state,
45+
used on 12 GB+ cards.
46+
- **Plain streaming** (~7.3 GB peak + 128 MB margin): per-plot
47+
allocations, no pinned-host parks, single-pass T2 match. ~400 ms/
48+
plot faster than compact. Used on 10-11 GB cards that can't fit
49+
the pool but have headroom above compact.
50+
- **Compact streaming** (~5.2 GB peak + 128 MB margin): full
51+
park/rehydrate + N=2 T2 match tiling. Used on 6-8 GB cards where
52+
plain won't fit. 6 GB cards (RTX 2060, RX 6600) are on the edge;
53+
8 GB cards (3070, 2070 Super) comfortably fit. Detailed breakdown
54+
in [VRAM](#vram).
4955
- **PCIe:** Gen4 x16 or wider recommended. A physically narrower slot
5056
(e.g. Gen4 x4) adds ~240 ms per plot to the final fragment D2H
5157
copy; check `cat /sys/bus/pci/devices/*/current_link_width`
@@ -326,6 +332,7 @@ batch — not a replacement for `chia plots check`.
326332
| Variable | Effect |
327333
|-------------------------------|-------------------------------------------------------------------------|
328334
| `XCHPLOT2_STREAMING=1` | Force the low-VRAM streaming pipeline even when the pool would fit. |
335+
| `XCHPLOT2_STREAMING_TIER=plain\|compact` | Override the streaming-tier auto-pick (plain = ~7.3 GB peak, no parks; compact = ~5.2 GB peak, full parks). |
329336
| `POS2GPU_MAX_VRAM_MB=N` | Cap the pool/streaming VRAM query to N MB (exercise streaming fallback).|
330337
| `POS2GPU_STREAMING_STATS=1` | Log every streaming-path `malloc_device` / `free`. |
331338
| `POS2GPU_POOL_DEBUG=1` | Log pool allocation sizes at construction. |
@@ -376,8 +383,8 @@ keygen-rs/ Rust staticlib: plot_id_v2, BLS HD, bech32m
376383

377384
## VRAM
378385

379-
PoS2 plots are k=28 by spec. Two code paths, dispatched automatically
380-
based on available VRAM:
386+
PoS2 plots are k=28 by spec. Three code paths, dispatched automatically
387+
based on available VRAM at batch start:
381388

382389
- **Pool path (~11 GB device + ~4 GB pinned host; 12 GB+ cards
383390
reliably).** The persistent buffer pool is sized worst-case and
@@ -387,9 +394,16 @@ based on available VRAM:
387394
`max(cap·12, 4·N·u32 + cub)` to `max(cap·12, 3·N·u32 + cub)` —
388395
saves ~1 GiB at k=28. Targets: RTX 4090 / 5090, A6000, H100,
389396
RTX 4080 (16 GB), and 12 GB cards like RTX 3060 / RX 6700 XT.
390-
- **Streaming path (5.2 GB peak + 128 MB margin; needs ≥ ~5.4 GiB
391-
*free* device VRAM at k=28).** Allocates per-phase and frees between
392-
phases. All three match phases (T1/T2/T3) are tiled N=2 across
397+
- **Plain streaming (~7.3 GB peak + 128 MB margin; ≥ 7.42 GiB free at
398+
k=28).** Allocates per-phase and frees between phases, but keeps
399+
large intermediates (`d_t1_meta`, `d_t1_keys_merged`, `d_t2_meta`,
400+
`d_t2_xbits`, `d_t2_keys_merged`) alive across their idle windows
401+
instead of parking them on pinned host. T2 match runs as a single
402+
full-cap pass (N=1). Used on 10-11 GB cards that can't fit the pool
403+
but have headroom above the compact floor. ~400 ms/plot faster than
404+
compact at k=28 because there are no park/rehydrate PCIe round-trips.
405+
- **Compact streaming (~5.2 GB peak + 128 MB margin; ≥ 5.33 GiB free
406+
at k=28).** All three match phases (T1/T2/T3) are tiled N=2 across
393407
disjoint bucket ranges with half-cap device staging and
394408
D2H-to-pinned-host between passes. T1 + T2 sorts are tiled (N=2 and
395409
N=4) with merge trees, and `d_t1_meta`, `d_t2_meta`, and the
@@ -415,23 +429,20 @@ based on available VRAM:
415429
Practical targets: 6 GB cards on the edge (card-dependent; RTX 2060
416430
typically has ~5.5 GiB free which has ~170 MB slack over the
417431
5328 MB requirement), 8 GB cards comfortable, 10 GB and up ample.
418-
Slower per plot (~3.7 s vs ~2.4 s at k=28 on a 4090) because it
419-
pays per-phase `malloc_device`/`free` plus ~2.5 GB of pinned-host
420-
round-trips for the parked-meta and T3 staging buffers, instead of
421-
amortising. Log the full alloc trace with `POS2GPU_STREAMING_STATS=1`.
432+
Log the full alloc trace with `POS2GPU_STREAMING_STATS=1`.
422433

423434
At pool construction `xchplot2` queries `cudaMemGetInfo` on the
424435
CUDA-only build, or `global_mem_size` (device total) on the SYCL
425436
path — SYCL has no portable free-memory query, so the check
426437
effectively approximates "free == total" and lets the actual
427-
`malloc_device` failure trigger the fallback. Either way, if the
428-
pool doesn't fit it transparently falls back to the streaming
429-
pipeline with no flag needed. Force streaming on any card with
430-
`XCHPLOT2_STREAMING=1`, useful for testing or for users who want
431-
the smaller peak regardless.
438+
`malloc_device` failure trigger the fallback. If the pool doesn't
439+
fit, the streaming-tier dispatch picks plain or compact based on
440+
the same free-VRAM query: plain if free ≥ 7.42 GiB, else compact.
441+
`XCHPLOT2_STREAMING=1` forces streaming even when the pool would
442+
fit; `XCHPLOT2_STREAMING_TIER=plain|compact` overrides the auto-pick.
432443

433-
Plot output is bit-identical between the two paths — the streaming
434-
code reorganises memory, not algorithms.
444+
Plot output is bit-identical across all three paths — streaming
445+
reorganises memory, not algorithms.
435446

436447
## Performance
437448

@@ -444,7 +455,8 @@ wall from `xchplot2 batch` (10-plot manifest, mean):
444455
| `cuda-only` branch | **2.15 s** | original CUDA-only path |
445456
| `main`, `XCHPLOT2_BUILD_CUDA=ON` (CUB sort) | 2.41 s | NVIDIA fast path on the SYCL/AdaptiveCpp port |
446457
| `main`, `XCHPLOT2_BUILD_CUDA=OFF` (hand-rolled SYCL radix) | 3.79 s | cross-vendor fallback (AMD/Intel) on AdaptiveCpp |
447-
| streaming path, ≤8 GB cards | ~3.7 s | pool path is preferred when VRAM allows |
458+
| plain streaming tier (10-11 GB cards) | ~5.7 s | no parks, single-pass T2 match; ~400 ms/plot faster than compact |
459+
| compact streaming tier (6-8 GB cards) | ~7.3 s | full parks + N=2 T2 match; minimum peak |
448460
| `main` on RX 6700 XT (gfx1031 / ROCm 6.2 / AdaptiveCpp HIP) | **9.97 s** | AMD batch steady-state at k=28; T-table AES near-optimal on RDNA2 via this compiler stack |
449461

450462
The `main`/CUB row is +12% over `cuda-only` from extra AdaptiveCpp

0 commit comments

Comments
 (0)