Skip to content

Commit 1da6437

Browse files
Abraham Sewillclaude
andcommitted
docs(spill): correct the wall cost, separate measured RSS from modelled
The ladder table and the tier table gave different numbers for the same k=28 compact config — 11.29 -> 4.24 GiB against 12.44 -> 5.33 — with nothing saying why. They measure different things: the ladder is peak RSS, the tier table is the modelled unswappable peak that `--max-host-ram` actually bounds. Both are now labelled, and the tier table carries an RSS column so the gap is visible rather than contradictory. `minimal` is the case that needs it: 9.19 GiB of RSS is 6.35 unswappable plus the 3.05 mapping. Wall cost was 9.71 -> 13.94 s/plot (~44%); it now measures 9.6 -> 12.6 (~31%, mean of 3) on the same card, partly because the drain-slot fix removes two 2.03 GiB pinned allocations. Adds the CPU-reference recipe: `test` without --gpu-all runs pos2-chip's own plotter, so the same plot built both ways must be byte-identical. That is the check that covers the streaming tiers and the disk-offload end to end, and nothing in the docs mentioned it. Verified verbatim. Also reorders the notes to match the SYCL branch — real disk, free space, traffic, what --max-host-ram bounds, unlinked files — so the two READMEs can be read side by side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 747f2c8 commit 1da6437

1 file changed

Lines changed: 37 additions & 11 deletions

File tree

README.md

Lines changed: 37 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -635,6 +635,23 @@ xchplot2 bench [-k K] [-n N] [-o DIR] [--devices <SPEC>] [--compute-only
635635
xchplot2 parity-check [--dir PATH] # CPU↔GPU regression screen
636636
```
637637

638+
To prove a plot correct, build the same one on the CPU and compare bytes.
639+
`test` without `--gpu-all` runs pos2-chip's own plotter, so the two must
640+
be byte-identical. The memo has to match — `batch` takes it from the
641+
manifest, `test` defaults to a 112-byte zero stub:
642+
643+
```bash
644+
MEMO=$(printf '00%.0s' $(seq 1 112))
645+
xchplot2 test 28 $PLOT_ID 2 0 0 -T -m "$MEMO" -o ref -N ref.plot2
646+
echo "28 2 0 0 1 $PLOT_ID $MEMO $PWD/out gpu.plot2" > m.tsv
647+
xchplot2 batch m.tsv --tier compact --max-host-ram min --temp-dir /mnt/nvme/spill
648+
sha256sum ref/ref.plot2 out/gpu.plot2
649+
```
650+
651+
The CPU side costs about a minute at k=28. Matching hashes clear the
652+
whole path at once — tier, disk-offload and GPU pipeline all agree with
653+
the reference implementation.
654+
638655
## Environment variables
639656

640657
| Variable | Effect |
@@ -872,28 +889,32 @@ budget requires:
872889
Measured at k=28 compact on an RTX 4090, one plot per rung — every rung
873890
produces a byte-identical plot (`sha256[:20] = 9e020867acd59d31164d`):
874891

875-
| routed | host peak |
892+
| routed | peak RSS |
876893
|---|---:|
877894
| nothing | 11.29 GiB |
878895
| `h_meta` | 9.32 GiB |
879896
| + `h_t2_xbits` | 8.30 GiB |
880897
| + drain 3 → 1 (`--max-host-ram min`) | **4.24 GiB** |
881898

882899
That is **2.66× less host RAM for the same plot**. The wall cost at the
883-
bottom rung, same card, temp dir on NVMe: **9.71 → 13.94 s/plot, ~44%**.
884-
4.24 GiB is the floor of this mechanism at k=28 — below it the remaining
885-
buffers are ones a GPU kernel writes directly through a device-visible
886-
pointer, which cannot live in a file.
900+
bottom rung, same card, temp dir on NVMe: **9.6 → 12.6 s/plot, ~31%**
901+
(mean of 3). 4.24 GiB is the floor of this mechanism at k=28 — below it
902+
the remaining buffers are ones a GPU kernel writes directly through a
903+
device-visible pointer, which cannot live in a file.
904+
905+
Note these are measured RSS. The budget line prints something different —
906+
the *modelled* peak of the unswappable class alone. Both are in the tier
907+
table below.
887908

888909
**Tier support is not uniform**, because what a tier does to a table
889910
decides whether the table can leave RAM at all:
890911

891-
| tier | how tables are routed | k=28 peak, `min` |
892-
|---|---|---:|
893-
| `plain` | drain slots only — it parks nothing ||
894-
| `compact` | both tables through the spill engine, to disk | 12.44 → 5.33 GiB |
895-
| `minimal` | both tables as file-backed **mappings** | 13.46 → 6.35 GiB |
896-
| `tiny` | drain slots only — no table can leave RAM | 14.47 → 10.41 GiB |
912+
| tier | how tables are routed | k=28 modelled peak, `min` | measured RSS at `min` |
913+
|---|---|---:|---:|
914+
| `plain` | drain slots only — it parks nothing || 7.20 GiB |
915+
| `compact` | both tables through the spill engine, to disk | 12.44 → 5.33 GiB | 4.24 GiB |
916+
| `minimal` | both tables as file-backed **mappings** | 13.46 → 6.35 GiB | 9.19 GiB |
917+
| `tiny` | drain slots only — no table can leave RAM | 14.47 → 10.41 GiB | 10.28 GiB |
897918

898919
`minimal` CPU-touches both tables, so they cannot go through the engine
899920
— but a `MAP_SHARED` mapping serves CPU indexing and `cudaMemcpyAsync`
@@ -935,6 +956,11 @@ Notes:
935956
sizes a drive's endurance: at 100 plots/day that is ~0.68 TiB/day,
936957
which consumes a 600 TB TBW rating in a little over two years. Point
937958
`--temp-dir` at something you are willing to wear out.
959+
- **`--max-host-ram` bounds the unswappable class** — pinned plus
960+
anonymous, the class that gets a process OOM-killed. On `minimal` both
961+
tables go as file-backed mappings instead: those bytes leave the
962+
dangerous class but stay resident until the kernel needs them back, so
963+
they still show in RSS. Where the two differ, the log reports both.
938964
- Files are unlinked at creation, so a crash cannot leave them behind,
939965
and each is `fallocate`d as it is created — a disk that fills anyway
940966
fails at once with its size rather than part-way through a table.

0 commit comments

Comments
 (0)