Follow steps 1–8 to serve Ouro-1.4B on one NVIDIA GPU and receive a generated response. Commands work in Bash and fish on Linux; use the activation command for your shell. Run them in the same terminal unless a step explicitly asks you to open another one.
This guide targets v0.2.0. Pin the tag below rather than a moving main.
For migration from v0.1.0, read the changelog; Python scripts
now construct a spawned stage worker and need the guarded example below.
The release is for research/evaluation, not a production deployment guarantee.
Use Linux with Git, curl, uv,
Python 3.10+, and an NVIDIA GPU
that supports BF16. Run nvidia-smi to confirm that the driver recognizes your
GPU. Installation and model download require access to GitHub, Python package
indexes, and Hugging Face. The BF16 model weights alone need roughly 3 GB of
GPU memory; leave additional room for the KV cache and runtime buffers.
uv venv --python 3.10 .venv # Replace .venv with your preferred environment name
source .venv/bin/activate # Bash; use the same environment name here
# fish: source .venv/bin/activate.fishPython 3.10 is an example; you can select another supported version (3.10+). Keep this environment active for the following steps.
git clone --branch v0.2.0 --depth 1 https://github.com/InferMatrixLab/vllm-rlt.git
cd vllm-rltSelect your allocated GPU before installation so automatic compilation targets the GPU you will use:
export CUDA_VISIBLE_DEVICES=0Use your allocated GPU ID in place of 0. If your job scheduler already sets
CUDA_VISIBLE_DEVICES, keep its value and skip the export.
uv pip install "torch>=2.5" setuptools wheel ninja
uv pip install --no-build-isolation -e .
python -m vllm_rlt.entrypoints.serve --helpThe help command should display the server options, confirming that the current environment can import the server entrypoint. It does not start the server.
This installs PyTorch, Transformers, the HTTP server dependencies, and Triton. The project requires PyTorch 2.5 or newer; it does not pin an exact version. uv keeps an existing compatible installation, or resolves a compatible version from your configured package index in a fresh environment.
Use the same installation and startup commands for GPUs such as the RTX 4090, A100, and H100. There is no CUDA version to select based on the GPU model; your NVIDIA driver must support the CUDA runtime supplied with PyTorch. FlashAttention and NIXL are optional and are not needed for this guide.
Confirm that PyTorch can use your GPU:
python -c "import torch; print('PyTorch:', torch.__version__, 'CUDA:', torch.version.cuda); assert torch.cuda.is_available(), 'CUDA unavailable'; assert torch.cuda.is_bf16_supported(), 'BF16 required'"Installation automatically builds Ouro's fused SiLU-and-Mul implementation when
it finds CUDA-enabled PyTorch, a CUDA toolkit with nvcc, and a visible GPU.
PyTorch finds the toolkit and detects the visible GPUs' target architectures;
no build switch or architecture setting is required. The NVIDIA driver and
PyTorch CUDA runtime alone do not provide the compiler; a compatible toolkit
and C++ compiler must also be installed.
If the requirements are missing, installation reports why the extension was skipped and retains the Torch implementation. To add the extension later, install the toolkit, keep your allocated GPU visible, and rerun:
uv pip install --no-build-isolation -e .This compiles and installs vllm_rlt._C using the environment's existing PyTorch.
Ouro selects the native kernel automatically for CUDA inputs; no server flag is
needed. Without the extension, it retains the Torch implementation. After changing
CUDA or C++ sources, rerun the build command and restart the server or Python
process. See native kernel development
for wheel builds, target GPU architectures and GPU validation commands.
Download the pinned checkpoint once, including tokenizer files. This separates network/download failures from server startup failures.
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
repo_id='ByteDance/Ouro-1.4B',
revision='574fa66cb8bf5abdc979642d01cf2b79b16bfab1',
local_dir='./artifacts/models/Ouro-1.4B',
allow_patterns=['*.json', '*.safetensors', '*.model', '*.txt'],
)
"Wait for the download to finish. Keep running commands from the repository
root so ./artifacts/models/Ouro-1.4B resolves to the same directory. For an existing
local checkpoint, replace this path in the commands below; it must contain
both the model configuration/weights and the tokenizer files. Loading uses
native code and safetensors, without trust_remote_code.
In the same terminal, run:
python -m vllm_rlt.entrypoints.serve \
--model ./artifacts/models/Ouro-1.4B \
--served-model-name ouro \
--device cuda \
--dtype bfloat16 \
--attention-backend triton \
--num-blocks 64 \
--max-num-seqs 1 \
--host 127.0.0.1 \
--port 8000Leave this process running. The equivalent installed command is
vllm-rlt-serve with the same arguments. Startup loads the model and tokenizer;
the first generation can also incur kernel compilation overhead.
This first-run configuration uses a bounded KV pool and one active sequence
for short prompts. It is not a performance-benchmark configuration. For longer
prompts or more concurrency, adjust cache capacity and scheduling as described
in the runtime guide. Omitting --num-blocks enables automatic
CUDA KV sizing.
Open a second terminal on the same machine. These commands only require curl; the server remains running in the first terminal.
curl -i http://127.0.0.1:8000/health
curl -sS http://127.0.0.1:8000/v1/modelsWait until /health returns HTTP 200. HTTP 503 means it is not ready;
inspect the first terminal for initialization progress or errors. The model
list must include ouro, matching --served-model-name.
Send a non-streaming request first:
curl --fail-with-body -sS http://127.0.0.1:8000/v1/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "ouro",
"prompt": "The capital of France is",
"max_tokens": 32,
"temperature": 0,
"exit_threshold": 1.0,
"stream": false
}'The first request uses fixed full depth (exit_threshold=1.0) to establish
the synchronous baseline. Lower thresholds enable adaptive exit and can change
output quality; the examples below demonstrate that optional behavior.
A successful response contains choices[0].text, a finish_reason, and usage
with prompt/completion token counts. The exact generated text can vary; checking
readiness alone does not verify generation.
To receive tokens as a stream:
curl --fail-with-body -N http://127.0.0.1:8000/v1/completions \
-H 'Content-Type: application/json' \
-d '{"model":"ouro","prompt":"The capital of France is","max_tokens":32,"temperature":0,"exit_threshold":0.7,"stream":true}'Expect SSE data: events ending with data: [DONE]. This is a completions
endpoint, not /v1/chat/completions. See the HTTP API reference
for supported fields, streaming behavior, and limits.
Press Ctrl+C in the server terminal to stop it and release its GPU resources. For a later session, return to the repository, reactivate the environment, select your allocated GPU, and rerun step 6. The model download does not need to be repeated.
cd /path/to/vllm-rlt
source ../.venv/bin/activate # Bash; use the environment name from step 2
# fish: source ../.venv/bin/activate.fish
export CUDA_VISIBLE_DEVICES=0Replace /path/to/vllm-rlt with your checkout path and preserve scheduler-set
GPU visibility when applicable. This completes the first-run path:
installation, model loading, readiness, a real completion, and shutdown.
For an offline batch, stop the HTTP server first if you will reuse the same GPU, then run from the activated environment and repository root:
python -m vllm_rlt.entrypoints.cli \
--model ./artifacts/models/Ouro-1.4B \
--device cuda \
--attention-backend triton \
--num-blocks 64 \
--max-num-seqs 1 \
--prompt 'The capital of France is' \
--prompt '2 + 2 =' \
--max-tokens 32 \
--exit-threshold 0.7The equivalent installed command is vllm-rlt. Repeat --prompt for a batch;
the CLI prints one JSON result per prompt, including text, token IDs, exit
depths, and the finish reason.
Use --attention-backend auto with the inference CLI or the local HTTP server
to select a compatible implementation at startup. Auto tries FA4, FA3, FA2,
Triton, then Torch, checking hardware, dtype, head dimension, block size, installed
packages, and execution requirements. CPU uses Torch. For example, on an RTX
4090, the default 16-token block size rules out FA2's 256-token page requirement;
auto selects Triton if it is installed.
GPU async scheduling and CUDA Graphs require a device backend. Packed prefill requires declared packed-prefill support and LAST_EXITED KV. Auto reports an error if no implementation satisfies these requirements. Selection occurs before memory profiling, and the startup log reports the selected backend and why earlier candidates were skipped.
--packed-prefill forces packed prefill and --no-packed-prefill disables it;
omitting both selects it automatically when supported. --no-uva independently
selects copies instead of mapped host-memory reads. See the
packed-prefill selection rules.
Explicit choices (torch, triton, flash_attn, or flash_attn_2/3/4) retain
their validation errors and never fall back. Existing defaults are unchanged:
the inference CLI defaults to Torch and the local server defaults to Triton.
The Python equivalent is attention_backend="auto". Read the worker startup
log for the selection decision. LLM.engine is a StageClient; worker-local
cache objects are not exposed through that client.
Save the following as example.py in the repository root, then run
python example.py from the activated environment. Stop the HTTP server first
if it uses the same GPU.
LLM starts a spawned worker process. In Python scripts, create it inside
main() and use the if __name__ == "__main__": guard shown below. The context
manager closes the worker when inference finishes or raises.
from vllm_rlt import CacheConfig, LLM, SamplingParams
def main():
with LLM(
"./artifacts/models/Ouro-1.4B",
device="cuda",
attention_backend="triton",
cache_config=CacheConfig(num_blocks=64),
) as llm:
outputs = llm.generate(
["The capital of France is", "2 + 2 ="],
SamplingParams(max_tokens=32, min_loops=2, max_loops=4, exit_threshold=0.7),
)
for output in outputs:
print(output.text)
print("Exit depths:", output.exit_depths)
if __name__ == "__main__":
main()Results preserve input order. Prompt prefill runs at full depth; the exit
threshold controls subsequent decode. Set exit_threshold=1.0 for fixed-depth
execution. The first generated token comes from full-depth prefill, reflected
in exit_depths. For dynamic arrivals, stepwise output, and cancellation, see
the architecture overview.
| Symptom | What to check |
|---|---|
vllm-rlt-serve: command not found |
Activate the environment from step 2 and rerun the editable install. The module entrypoint python -m vllm_rlt.entrypoints.serve uses the same server. |
No module named vllm_rlt, aiohttp, or transformers |
Activate the environment from step 2, verify sys.executable, and rerun uv pip install -e .. |
| CUDA check fails or reports a driver error | Check your GPU allocation and CUDA_VISIBLE_DEVICES. If the driver is too old for the installed PyTorch CUDA build, update the driver or choose a compatible build using the PyTorch installer. The installer does not select builds based on your installed driver. See NVIDIA driver compatibility. |
| Model download fails | Check access to Hugging Face and available disk space, then rerun step 5. An offline deployment needs a complete downloaded checkpoint and tokenizer. |
| CUDA out of memory | Stop duplicate model processes you started, inspect free memory with nvidia-smi, and use the bounded short-prompt configuration in step 6. The weights and runtime still need sufficient free memory. |
/health gives 503 or connection refused |
Check the server terminal. It may still be initializing or may have exited with an error. |
| Port 8000 is already in use | Stop your old server or choose another --port and use it in all curl URLs. |
| Request returns an unknown-model error | The JSON model must match --served-model-name: ouro in this guide. |
| Request is rejected for KV capacity/context limits | Shorten the prompt/output or increase the KV budget for your workload. The first-run cache is intentionally small. |
| Chat endpoint returns 404 | Use /v1/completions with a string prompt. |
When reporting a problem, include the failing command, server error, GPU model,
PyTorch/CUDA versions from step 4, and your Git revision (git rev-parse HEAD).
Prompt chunks normally execute their full recurrence depth in one submission.
On supported models with LAST_EXITED KV, --mode mixed lets the scheduler
insert a depth boundary when decode work can join a prompt recurrent batch.
--wavefront-chunked-prefill enables per-depth chunk regrouping. These are
batching choices, not PD disaggregation.
Stage roles are prefill, decode and prefill_decode; mixed is a scheduling
mode, not a stage role. Speculation requires refill scheduling and cannot join
mixed recurrent batches. Model/backend restrictions still apply. See the
runtime guide before
combining optional features.