Skip to content

Latest commit

 

History

History
58 lines (51 loc) · 4.43 KB

README.md

File metadata and controls

58 lines (51 loc) · 4.43 KB

Documentation — v0.2.0

This version is a research/evaluation milestone. Start with synchronous execution; optional GPU paths and historical measurements have narrower qualification scopes than the feature list. See the changelog and migration notes before upgrading from v0.1.0.

New users should follow the first-run user guide, from creating an environment to receiving the first generated response. The README provides a brief installation reference.

Guide Topics
Developer Must-Read Requirement understanding, correctness, performance analysis, refactoring rules, and completion criteria
Native kernel development Optional CUDA extension, operator layout, and SiLU-and-Mul reference contract
First-run user guide Environment setup, model download, server startup, first request, CLI, Python API, and troubleshooting
Unified architecture Stage process boundaries, component responsibilities, shared resources and loop-level execution
Model and KV semantics PRELUDE/RECURRENT/CODA, request progress and depth-specific history
Loop-level scheduling Mixed batching, policy selection, admission budgets, progress and resource safety
KV layout examples SHARED and LAST_EXITED semantics and worked attention examples
KV cache walkthrough Capacity, validity, shared metadata, transfer pins and snapshot/restore
Runtime execution and configuration Batch contracts, preparation, transfers, execution options and compatibility
Split prefill and WCPB Chunk/depth scheduling, hidden-state ring, KV dependencies and validation limits
Self-speculation Cross-round asynchronous block execution, token pipelines, and acceptance/rejection timelines
Asynchronous scheduling CPU/GPU pipelining and single-stream or multi-stream execution
FlashAttention FA2/FA3/FA4 installation, hardware selection, and constraints
Cache and scheduling features Prefix reuse, incremental KV, priorities, and preemption
Prefill/decode disaggregation Single-host GPU worker pools and NIXL transfer
HTTP serving Completions API, streaming, and service lifecycle
Profiling Core profiler options, Python/HTTP controls, PD collection, and per-rank archives
Ouro-Thinking recipe Shared 1.4B/2.6B setup, JSON/SSE client, budgets and validation limits
Accuracy evaluation GSM8K regression setup and comparison methodology

Earlier measurements and known limitations are retained in the historical validation record; they do not qualify the current revision.

Support and Runtime Notes

Supported models are Ouro-1.4B, Nanbeige4.2-3B, Huginn-0125 and NVIDIA-Nemotron-Labs-Dory1.0-4B. See the checkpoint support matrix and the Thinking recipe for model-specific coverage. Huginn currently uses synchronous execution, LAST_EXITED KV, and fixed recurrent depth; prefix caching, asynchronous execution, and speculative decoding are rejected, as are mixed/WCPB staged prefill and PD. Its CUDA Graphs capture the recurrent decode core and coda, including the LM head. The Dory baseline notes list the narrower scope qualified for Dory: synchronous fixed-loop execution with boundary KV in both KV modes, sliding-window masking on the Torch and Triton backends, and BF16 agreement with the pinned checkpoint at 1/2/4 loops; Graphs, WCPB, async, speculation, prefix reuse and PD remain out of scope. CPU execution provides a Torch reference backend. FlashAttention hardware validation is currently documented for FA4 on B300; FA2/FA3 require validation on their target devices. Disaggregated serving currently targets multiple GPUs on a single host.

ouro_delayed reuses the trained Ouro gate with a one-loop delay; it changes the exit policy. The random_lookahead mode uses an untrained head for runtime experiments. Neither is a distilled lookahead predictor.