Skip to content

GPU.md

Latest commit

 

History

History
274 lines (235 loc) · 17.3 KB

GPU.md

File metadata and controls

274 lines (235 loc) · 17.3 KB
Raw

GPU — the in-kernel forward pass, on the GPU (on-box witness + honest gap to llama.cpp)

This is the doc the repo's "no GPU claim without an on-box witness" rule (IN-KERNEL-MODEL-DESIGN.md) was waiting for. GPU was SIMULATED / forbidden to claim. It now isn't: a modular CUDA backend behind the frozen compute.Backend seam runs a real Llama decode forward pass on this box's RTX 4070, witnessed against the CPU reference, and — with a reusable CUDA graph — decodes at parity with llama.cpp Q8_0 (≈120 tok/s) on a model that fits the GPU (§3b). Every number below is from a real run captured in the test/bench output, not asserted; the one honest caveat (the parity path is gated + uses a fixed-capacity KV) is stated just as plainly.

0. What shipped (this lane)

Piece Where Status
CUDA backend (cgo + cuBLAS + custom kernels) implementing compute.Backend internal/compute/cuda.go, cuda_kernels.cu, cuda_backend.h shipped
Registered as Approx peer "cuda"; cpu-ref stays the Reference Default cuda.go:init shipped
Device-resident KV cache (VRAM K/Kraw/V; on-device AppendKV/Clone) cuda.go:cudaKV shipped (Evict via host round-trip, deferred → #39)
Model leaf runs decode through the HAL — same model on CPU (bit-exact) or GPU (Approx) internal/model/hal.go shipped
Reusable CUDA graph (cudaGraphExecUpdate) + kernel KV-append + cross-session weight share → decode parity with llama.cpp Q8_0 (≈120 tok/s) cuda_kernels.cu, cuda.go shipped, gated FAK_CUDA_GRAPH=1 (§3b)
No-sudo WSL CUDA toolchain + two-step nvcc→static-lib→cgo build setup_cuda_wsl.sh, build_cuda.sh shipped
Default go build stays one pure-Go artifact; CUDA is opt-in -tags cuda — verified

1. The on-box witness (real runs, RTX 4070 Laptop, sm_89, CUDA 12.6, WSL2)

The toolchain itself is witnessed first — nvcc compiles an sm_89 kernel and it executes on the GPU (setup_cuda_wsl.sh → DEVICE NVIDIA GeForce RTX 4070 Laptop GPU sm_89, RESULT y[0]=5.0). Then the backend:

Witness Result Test
Device SGEMM vs cpuref fdot matmul cosine 1.00000000, maxAbs 1.91e-06 TestCUDAMatMulApproxMatchesRef
Full 3-layer Llama decode, 6 prompt + 8 greedy tokens, GPU vs cpuref argmax-exact every step, final logit cosine 1.0 TestCUDAForwardMatchesRef
Real in-kernel model decode on the GPU vs native path argmax-exact over 10 tokens, prefill cosine 1.0 TestHALDeviceForwardMatchesNative
HAL adoption preserves the proven path on cpuref Float32-bit-identical logits + token-for-token greedy TestHALPrefillLogitsMatchNativeBitExact, TestHALGenerateMatchesNativeBitExact

The last row is the load-bearing safety property: routing the model through the HAL is byte-for-byte identical on the Reference backend (so R2/R14 and the HF oracle cannot be perturbed), while the same code runs Approx on the device. compute.RequireReference keeps the device off the bit-identity rungs by construction.

Reproduce from fak/: bash internal/compute/setup_cuda_wsl.sh once, then bash internal/compute/build_cuda.sh test. From the repo root, the laptop lane runner wraps the same path and also exposes the CPU/Intel lane. On Anthony's Windows laptop, use the PowerShell wrapper:

.\tools\fak_laptop_test.ps1 accept
.\tools\fak_laptop_test.ps1 accept --full-cpu   # same acceptance path, but CPU runs ./...
.\tools\fak_laptop_test.ps1 accept --cpu-only   # CPU/Intel proof, no NVIDIA required

If the usual checkout has unrelated dirty work and git pull would be risky, run the proof from a detached clean worktree instead:

git fetch origin
git worktree add --detach ..\fleet-laptop-proof origin/master
cd ..\fleet-laptop-proof
.\tools\fak_laptop_test.ps1 accept

That single command runs the required laptop acceptance sequence: CPU/NVIDIA passthrough preflight, CPU smoke, CUDA setup, CUDA tests, post-setup toolchain checks, and report verification. Use --full-cpu when the CPU/Intel lane needs the full ./... suite instead of the focused smoke tests. Use --cpu-only when validating the CPU/Intel lane on its own; it writes fak\experiments\gpu\laptop-cpu-check.json and fak\experiments\gpu\laptop-cpu.json, then verifies those artifacts without requiring NVIDIA passthrough. To isolate a failure, run the lower-level steps directly:

.\tools\fak_laptop_test.ps1 check --require-nvidia --out fak\experiments\gpu\laptop-check.json
.\tools\fak_laptop_test.ps1 cpu --smoke            # focused CPU/HAL smoke
.\tools\fak_laptop_test.ps1 cpu -- ./...           # full CPU suite
.\tools\fak_laptop_test.ps1 check --out fak\experiments\gpu\laptop-cpu-check.json
.\tools\fak_laptop_test.ps1 cpu --smoke --out fak\experiments\gpu\laptop-cpu.json
.\tools\fak_laptop_test.ps1 cpu --out fak\experiments\gpu\laptop-cpu.json   # full CPU report
.\tools\fak_laptop_test.ps1 nvidia --setup --out fak\experiments\gpu\laptop-nvidia.json
.\tools\fak_laptop_test.ps1 check --require-nvidia --require-cuda-toolchain --out fak\experiments\gpu\laptop-post-setup.json
.\tools\fak_laptop_test.ps1 all --smoke --setup --out fak\experiments\gpu\laptop-all.json
.\tools\fak_laptop_test.ps1 verify
.\tools\fak_laptop_test.ps1 verify --cpu-only
.\tools\fak_laptop_test.ps1 verify --cpu-only --full-cpu
.\tools\fak_laptop_test.ps1 status
.\tools\fak_laptop_test.ps1 status --cpu-only

On Linux/macOS, call the same runner as python tools/fak_laptop_test.py ...; the NVIDIA lane intentionally refuses to run on macOS except in --dry-run mode. Use --wsl-distro NAME or FAK_WSL_DISTRO=NAME to pin a distro; otherwise the Windows runner matches fak/test.ps1 by preferring Ubuntu-24.04 when it is installed and falling back to WSL's default distro. On Windows, both the CPU and NVIDIA lanes default to the WSL ext4 mirror fast path (FAK_FAST=1) and the reports record wsl.filesystem=ext4-mirror; use --no-fast or FAK_FAST=0 only when intentionally auditing the slower /mnt/c drvfs path. The accept lane writes fak\experiments\gpu\laptop-check.json, fak\experiments\gpu\laptop-all.json, and fak\experiments\gpu\laptop-post-setup.json; relative report paths are resolved from the repo root even when the wrapper is launched from another directory. The verify lane reads the post-setup check and all-lane reports by default and fails unless CPU, NVIDIA passthrough, the CUDA toolchain, CUDA setup, and CUDA tests all passed. CPU check reports also record the Go target (GOOS/GOARCH) so the laptop artifacts show which CPU lane was actually tested. Reports include proof metadata (mode and, for run reports, cpu_scope) so verify --cpu-only rejects NVIDIA/CPU artifacts and verify --full-cpu rejects smoke-only CPU runs. Reports also include git revision and dirty-worktree summary metadata, including a bounded dirty-path sample, so laptop artifacts can be tied back to the exact repo state that produced them. Each report also has a compact summary block for quick pass/fail review. Use status to print those summaries, proof metadata, repo metadata, and per-check / per-command results from the existing report files without rerunning any tests; it exits non-zero if the selected reports are missing, from the wrong proof mode, failed, or only dry-run artifacts. verify and status also reject reports from a different git HEAD or dirty-worktree fingerprint by default; add --allow-stale-repo only when deliberately inspecting older artifacts.

2. Why this is "native as reasonable" on THIS box

  • WSL2 + CUDA is real CUDA on the real GPU. WSL2 GPU passthrough is live (nvidia-smi sees the 4070 from Ubuntu, /usr/lib/wsl/lib/libcuda.so present); the kernels are compiled by a real nvcc for sm_89 and run on the Ada silicon. The only thing "virtual" is the driver shim, which is how essentially all Linux-on-Windows GPU compute works.
  • It sidesteps the WDAC policy that blocks unsigned native-Windows binaries from %TEMP% — the same reason the Go test suite already runs in WSL (CLAUDE.md). A native Windows -tags cuda build (signed) is the portability follow-up (#37).
  • No sudo, no system mutation. The toolkit is a user-space micromamba CUDA 12.6 env; the default go build needs none of it and stays pure-Go.
  • It honors DIRECTION.md. CUDA C++ is a sanctioned hardware seam in a statically typed compiled language, off the request path, behind the re-validated typed boundary (the flat C ABI carries device pointers + shapes, never trust). The default artifact is still one pure-Go binary (GoFiles=[compute.go cpuref.go]; cuda.go is a CgoFile only under -tags cuda).

3. Parity baseline — llama.cpp CUDA on this 4070 (measured)

The model is Qwen2.5-7B-Instruct Q4_K_M — the GPU parity target chosen in GPU-MODEL-PICK.md (strongest agentic model that fully fits 8 GB; weights ~4.68 GB, ~1.6–1.8 GiB headroom at 4–8K ctx). llama.cpp is built with CUDA from source in the same WSL+CUDA environment the fak backend uses, so the GPU + passthrough are identical.

Measured (RTX 4070 Laptop, 8188 MiB / 7090 free, sm_89, CUDA 12.6, WSL2; llama-bench -ngl 99 full GPU offload, median of 5 — a cold first invocation read low and was discarded):

test t/s
prefill pp512 2256 ± 45
prefill pp256 2245 ± 73
prefill pp64 1432 ± 298
decode tg128 48.0 ± 0.4
decode tg64 46.3 ± 1.0

Weights resident: 4.36 GiB on the ~7.0 GiB free budget — the GPU-MODEL-PICK.md fit confirmed on real hardware. Reproduce (after setup_cuda_wsl.sh and a CUDA llama.cpp build): llama-bench -m Qwen2.5-7B-Instruct-Q4_K_M.gguf -ngl 99 -p 512 -n 128 -r 5.

This is the bar — and the honest asymmetry must be stated. fak's Go-CUDA backend cannot yet run Qwen2.5-7B-Q4_K_M end-to-end, so there is no fak-vs-llama tok/s at 7B in this session, by construction: the in-kernel loader holds the whole f32 blob (a 7B f32 set is ~28 GB and will not fit WSL's ~15 GB RAM), and there is no GGUF / Q4_K device path yet. A 7B head-to-head is gated on the loaders (GGUF + quant-on-load) plus quantized device GEMM, tracked in the issue tracker. What this session delivers is the baseline to beat (above) and a witnessed modular GPU path (§1) to grow toward it — not a parity claim fak has not yet earned. (For the CPU axis, the existing LLAMACPP-HEADTOHEAD-RESULTS.md already measures fak at-best-parity / otherwise-behind llama.cpp on SmolLM2-135M; this GPU doc is the orthogonal GPU axis.)

3b. The head-to-head fak CAN run today — SmolLM2-135M, GPU, same box

The 7B is loader-gated, but fak does run SmolLM2-135M on the GPU today (the proven in-kernel checkpoint), so this is a real fak-vs-llama.cpp head-to-head on a model that fully fits the GPU — measured on the same RTX 4070 / WSL+CUDA, decode median over 128 steps.

engine precision decode t/s prefill t/s (P=64)
fak-CUDA, op-per-call (lean path) f32 7.5 24
fak-CUDA, reusable CUDA graph (FAK_CUDA_GRAPH=1) f32 119–120 ~125
llama.cpp Q8_0 120 ± 15 —
llama.cpp F16 261 ± 10 —

Decode throughput parity REACHED. With the reusable-graph path, fak-CUDA decodes at ~120 tok/s — dead even with llama.cpp Q8_0 (120 tok/s) on a model that fits the GPU, and at higher precision (fak runs f32; llama.cpp Q8_0 is 8-bit, so fak moves 4× the weight bytes per token and still matches it). Against llama.cpp F16 (261, 2-byte weights) fak f32 is ~46% — the residual is the f32-vs-f16 memory-bandwidth difference, the lever issue #34 (fp16) addresses, not an architecture gap. Output parity also holds (greedy argmax-exact vs the reference). Getting here was a 16× decode speedup (7.5 → 120 tok/s).

Step 1 — the diagnosis (a microbench of the bare WSL CUDA floor). fak's compute is trivial (a 135M token is ~270 MFLOP, microseconds on a 4070); the cost was CUDA-API launch/submission overhead on WSL. Timing the bare floor (N kernel launches + N cuBLAS GEMVs, one device sync at the end):

op (WSL, async submit) cost
bare kernel launch 0.071 ms
kernel launch + sync 0.283 ms
cuBLAS sgemm GEMV (1×2048×2048) 0.134 ms

⇒ the ~210 cuBLAS GEMVs/token alone floor an op-per-call decode at ~80 ms ≈ 12 tok/s, independent of compute. So the op-per-call backend cannot reach llama.cpp on WSL — the fix had to collapse the ~600 host CUDA calls/token into ~one. (Cleaning the non-compute overhead — resident weight cache, pooled allocator + recycling, async copies, single stream — first took it 2.7 → 7.5 t/s and confirmed the residual was the launch count.)

Step 2 — the dead end. Per-token cudaStreamBeginCapture/Instantiate/Launch did NOT help (7.0 t/s): re-instantiating a ~600-node graph every token costs ~what the 600 launches cost.

Step 3 — the win: a REUSABLE graph (instantiate-once). Three changes made one captured graph replayable across the whole growing decode, so the per-token cost is one launch:

  1. cudaGraphExecUpdate — keep the instantiated exec; each token capture a fresh graph (identical topology) and patch the changed params (position, nPos, KV offset) into the kept exec instead of recompiling.
  2. kernel-form KV-append (k_copyrow, scalar offset) replacing the cudaMemcpy whose destination pointer grew every token — a moving pointer ExecUpdate could not patch, which was silently forcing a re-instantiate on every decode token.
  3. cross-session weight sharing (Upload caches device buffers by host pointer) — without it every session re-uploaded the whole model and VRAM exhausted mid-bench, which had been masking the decode result entirely.

Result: 7.5 → 119–120 tok/s decode (16×), at parity with llama.cpp Q8_0. It ships gated FAK_CUDA_GRAPH=1 with one honest caveat: the device KV is fixed-capacity (1024 positions) so capture never hits a cudaMalloc; lifting that to dynamic/ring is the follow-up (dynamic/ring device KV), and fp16 is the lever to also reach the F16 number. Native-Linux (no WSL per-call tax) would start from a far lower floor.

Reproduce: fak FAK_CUDA_GRAPH=1 go run -tags cuda ./cmd/modelbench -dir internal/model/.cache/smollm2-135m -backend cuda (or internal/compute/build_cuda.sh bench); llama llama-bench -m SmolLM2-135M-Instruct-{f16,Q8_0}.gguf -ngl 99 -n 128.

4. What's still open (decode parity reached; these extend it)

Decode parity with llama.cpp Q8_0 is reached on a fitting small model (§3b). These follow-ups extend it — to the F16 number, to a 7B, and to a non-gated default — tracked in the issue tracker:

Gap vs llama.cpp Consequence
No Q4_K device path Q4_K GPU throughput vs llama.cpp (Q8 shipped in v0.13.0; F16 shipped in v0.13.0)
Naive decode attention (per-call scratch, one block/head) no flash/paged attention; launch + memory overhead
Synchronous, one kernel launch per op batch-1 decode is launch-bound (async + CUDA Graphs)
No GGUF / sharded / quant-on-load the 7B Q4 target can't be loaded by the in-kernel path yet (28 GB f32 won't fit 15 GB WSL RAM)
Evict round-trips host-ward quarantine correct but slow on device
7 hand-copied blocks not yet consolidated arch-dispatch + clean HAL adoption blocked (SEAM-0)

The umbrella that closes the loop — a measured Go-CUDA tok/s next to the llama.cpp baseline on Qwen2.5-7B-Q4_K_M, same batch-1 protocol — is tracked in the issue tracker.

5. Bottom line

  • GPU is no longer SIMULATED. A modular CUDA backend behind compute.Backend runs the in-kernel model's decode on the RTX 4070, witnessed argmax-exact vs the CPU reference, with the HAL adoption proven byte-identical on the Reference path.
  • Modularity held the line. Adding the GPU was a new Backend registration + a build tag; the forward loop and the default pure-Go artifact were untouched, exactly as the HAL was designed for.
  • Decode throughput parity: REACHED. On SmolLM2-135M (a model that fully fits the GPU), fak-CUDA decodes at 119–120 tok/s — even with llama.cpp Q8_0 (120) — and at higher precision (fak f32 vs llama Q8_0), with output (greedy) parity too. That is a 16× jump (7.5 → 120 tok/s) this session, via a reusable CUDA graph: cudaGraphExecUpdate (instantiate-once), a kernel-form positioned KV-append (so the graph stays patchable as the cache grows), and cross-session weight sharing (so VRAM stops exhausting). The diagnosis that drove it was a microbench proving the op-per-call path is launch-bound on WSL (~80 ms/token floor) — the kernel-program boundary, now crossed. vs llama.cpp F16 (261) fak f32 is ~46%, an f32-vs-f16 bandwidth gap (#34), not architecture.
  • Honest caveat: the parity path is gated FAK_CUDA_GRAPH=1 and uses a fixed-capacity device KV (1024 positions, so capture never hits a cudaMalloc); making it dynamic and default-on is the follow-up (#39). 7B-on-GPU is still loader-gated (#40/#41). Nothing here is claimed beyond what the committed modelbench + llama-bench numbers show.