Motivation.
Speculators can generate hidden states online for training, but it does not currently
provide a bounded shared data plane for multiple independent trainer processes. In a
1-producer/3-consumer topology, issuing one producer request per consumer repeats
prefill work. Driving all consumers from one synchronous cursor instead makes a
heterogeneous group advance at the pace of its slowest member.
The goal is to let each single-GPU consumer advance at its own rate while reusing a
producer result whenever consumer windows overlap. Retention must be bounded by the
active windows rather than by the length of a 100k-step or longer training run.
This is intentionally independent-process fan-out, not tensor or data parallelism:
each consumer has its own model, optimizer, cursor, and GPU.
Proposed Change.
Add an opt-in bounded asynchronous artifact path around the existing online training
flow:
- A filesystem artifact cache fingerprints a hidden-state request, coalesces
concurrent misses, and publishes a packed result atomically. Existing online and
offline paths remain unchanged unless the new CLI options are supplied.
- Each consumer registers an independent cursor and a lookbehind/lookahead window.
The trainer process is the only authority that commits cursor progress; DataLoader
workers may acquire authorized positions but cannot move the window.
- Artifacts remain reusable while they belong to the union of live consumer windows.
Publications outside every window are evicted after active leases and packed-batch
references are released. A lagging consumer may regenerate an evicted item instead
of forcing faster consumers to wait.
- A bounded asynchronous scheduler prioritizes demand misses and fills lookahead in
the background. Queue, in-flight, and producer capture-batch limits provide
backpressure, so memory and producer work cannot grow with total training steps.
- Failure state is explicit and retryable. Cleanup, publication, lease, timeout, and
recapture accounting is exposed for validation rather than silently treating a
partial artifact as a cache hit.
- A benchmark harness runs fresh 1P1C and 1P3C scenarios serially, rejects distributed
launchers or shared GPUs, validates logical-request/service-completion accounting,
and records a common post-warmup interval with role-aware NVML telemetry.
The initial data plane uses a shared POSIX filesystem and therefore requires reliable
flock, same-filesystem atomic rename, and directory fsync behavior. It is not a
claim of GPU-direct or arbitrary-NFS support. The cache/coordinator boundary is kept
separate so another transfer backend can be considered later.
The proposed branch also includes opt-in DFlash consumer improvements needed to avoid
moving the bottleneck from the producer to avoidable optimizer and loss overhead:
fused AdamW with integrated gradient clipping, and a pinned optional Liger fused linear
cross-entropy path for memory-constrained runs. The default training behavior is
preserved.
Any Other Things.
The state-machine tests exercise 10k- and 100k-position streams and verify that retained
state is bounded by active windows. Focused affected tests currently report 181 passed
and 1 skipped; the compiled optional Liger CUDA forward/backward test passed separately.
Quality checks pass on Python 3.10, 3.12, and 3.13.
In three aligned Qwen3-8B DFlash 1P3C runs (3072-token sequences, 256 anchors, block
sizes 4/8/16, 30 warmup and 150 measured steps per consumer), aggregate steady consumer
throughput was 6.3735, 6.2856, and 6.5702 steps/s. The median per-consumer rates were
2.2643, 2.3135, and 1.6978 steps/s. Producer utilization reached 100% over the common
steady interval, showing that the consumers advance independently and that producer
capacity, rather than a synchronous consumer barrier, is the remaining limit in this
configuration.
I would like maintainer feedback on the cache/window contract and filesystem scope
before opening the implementation PR.
Motivation.
Speculators can generate hidden states online for training, but it does not currently
provide a bounded shared data plane for multiple independent trainer processes. In a
1-producer/3-consumer topology, issuing one producer request per consumer repeats
prefill work. Driving all consumers from one synchronous cursor instead makes a
heterogeneous group advance at the pace of its slowest member.
The goal is to let each single-GPU consumer advance at its own rate while reusing a
producer result whenever consumer windows overlap. Retention must be bounded by the
active windows rather than by the length of a 100k-step or longer training run.
This is intentionally independent-process fan-out, not tensor or data parallelism:
each consumer has its own model, optimizer, cursor, and GPU.
Proposed Change.
Add an opt-in bounded asynchronous artifact path around the existing online training
flow:
concurrent misses, and publishes a packed result atomically. Existing online and
offline paths remain unchanged unless the new CLI options are supplied.
The trainer process is the only authority that commits cursor progress; DataLoader
workers may acquire authorized positions but cannot move the window.
Publications outside every window are evicted after active leases and packed-batch
references are released. A lagging consumer may regenerate an evicted item instead
of forcing faster consumers to wait.
the background. Queue, in-flight, and producer capture-batch limits provide
backpressure, so memory and producer work cannot grow with total training steps.
recapture accounting is exposed for validation rather than silently treating a
partial artifact as a cache hit.
launchers or shared GPUs, validates logical-request/service-completion accounting,
and records a common post-warmup interval with role-aware NVML telemetry.
The initial data plane uses a shared POSIX filesystem and therefore requires reliable
flock, same-filesystem atomic rename, and directoryfsyncbehavior. It is not aclaim of GPU-direct or arbitrary-NFS support. The cache/coordinator boundary is kept
separate so another transfer backend can be considered later.
The proposed branch also includes opt-in DFlash consumer improvements needed to avoid
moving the bottleneck from the producer to avoidable optimizer and loss overhead:
fused AdamW with integrated gradient clipping, and a pinned optional Liger fused linear
cross-entropy path for memory-constrained runs. The default training behavior is
preserved.
Any Other Things.
The state-machine tests exercise 10k- and 100k-position streams and verify that retained
state is bounded by active windows. Focused affected tests currently report 181 passed
and 1 skipped; the compiled optional Liger CUDA forward/backward test passed separately.
Quality checks pass on Python 3.10, 3.12, and 3.13.
In three aligned Qwen3-8B DFlash 1P3C runs (3072-token sequences, 256 anchors, block
sizes 4/8/16, 30 warmup and 150 measured steps per consumer), aggregate steady consumer
throughput was 6.3735, 6.2856, and 6.5702 steps/s. The median per-consumer rates were
2.2643, 2.3135, and 1.6978 steps/s. Producer utilization reached 100% over the common
steady interval, showing that the consumers advance independently and that producer
capacity, rather than a synchronous consumer barrier, is the remaining limit in this
configuration.
I would like maintainer feedback on the cache/window contract and filesystem scope
before opening the implementation PR.