Skip to content

CLAUDE.md

Latest commit

 

History

History
95 lines (68 loc) · 19.5 KB

CLAUDE.md

File metadata and controls

95 lines (68 loc) · 19.5 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

Two overlapping things live here:

  1. A content repo for vLLM recipes — structured YAML at models/<hf_org>/<hf_repo>.yaml, one per model. The path mirrors HuggingFace so the site URL /<hf_org>/<hf_repo> matches huggingface.co/<hf_org>/<hf_repo>.
  2. A Next.js 15 site (App Router, React 19, Tailwind v4) that renders those YAMLs as an interactive command builder + static JSON API under public/. The older top-level Markdown directories (DeepSeek/, Qwen/, etc.) and mkdocs.yml are the legacy MkDocs site, kept for reference during migration.

The site is the canonical surface. New work goes into YAMLs + the Next.js app; the .md files are historical sources for the migrated YAMLs.

Commands

pnpm dev                                # sync + assemble sources, then the Next dev server (do not kill or rebuild as part of a task)
pnpm build                              # Regenerates JSON API + fetches logos/HF dates, then next build
pnpm lint                               # next lint (ESLint)
node scripts/build-recipes-api.mjs      # Assemble content + rebuild the JSON API — fastest validation that all YAMLs parse
node scripts/sync-sources.mjs [--update] # Fetch remote recipe sources pinned in sources.lock.json (--update re-resolves refs)
node scripts/assemble-content.mjs [--strict] [--override name=dir]  # Merge a channel's sources into .content/<channel>/
pnpm content:watch                      # Re-assemble when files are added/removed in a local source (alongside pnpm dev)
pnpm content:update                     # Re-resolve every source ref and rewrite sources.lock.json
node scripts/validate-source.mjs --source <name> --dir <path>  # Validate a source repo's recipe dir (what its PR check runs)
pnpm test:overlay                       # Overlay merge/split round-trip over every recipe + scope violations
pnpm test:schema                        # Recipes + templates match the schema; misspelled keys are rejected
node scripts/fetch-provider-logos.mjs   # Re-download HF org avatars to public/providers/
node scripts/fetch-hf-dates.mjs         # Refresh public/hf-dates.json (HF createdAt per model)
bash scripts/hf-info.sh <org>/<repo>    # Pull config.json / params.json for a model (used when authoring recipes)

There is no unit-test suite. The build-recipes-api script is the de-facto validator: if it prints ✓ JSON API: N models, <strategy-count> strategies the YAML set is internally consistent.

Dev-server convention: don't kill or restart the dev server as part of a task, and don't run pnpm build for each change. Use node scripts/build-recipes-api.mjs for fast feedback on YAML edits; rely on Next's HMR for src/ edits.

Commit convention: always use git commit -s (sign-off). The upstream vllm-project/recipes repo enforces DCO via the probot/dco GitHub App — commits without a Signed-off-by: trailer fail CI and block the PR. There's no GitHub-side auto-fix; the trailer has to be in the commit itself.

Architecture

Content sources

Recipe YAMLs are moving out of this repo into the repos that own them — vllm (core models), vllm-omni (omni recipes), vllm-ascend (Ascend overlays) — with this repo as the renderer that also owns the vocabulary (taxonomy / strategies / kv_store / platforms). sources.yaml names each channel's sources (repo, ref, path of the dir holding models/ + overlays/, role: core|owner|overlay, scope, or local: for a checkout); sources.lock.json pins remote ones to a commit (--update only bumps when the recipe dir's tree_sha changes). scripts/assemble-content.mjs merges a channel into .content/<channel>/ with the old repo-root layout (unmerged files symlinked locally, copied on CI) plus provenance.json (hf_id → repo/sha/path/edit_url/overlays). Every loader reads through contentDir() in src/lib/content-root.js (RECIPES_CONTENT_DIR > .content/<RECIPES_CHANNEL or default_channel> > repo root), and recipe objects / JSON carry it as source. build-recipes-api.mjs assembles first, so new YAMLs in any local source are picked up. Today latest = vllm (core, recipes/ of vllm-project/vllm@main, every non-omni recipe) + this repo's own models/ (local: ., interim owner of the omni recipes until they move to vllm-omni). models/ also still holds copies of the vLLM recipes until they are batch-synced to vllm and removed: defers_to: vllm skips every copy vllm also ships, and a recipe only here (omni, or one merged here that vllm doesn't have yet) renders from here. Ascend content still lives inline in the vllm YAMLs until vllm-ascend ships overlays. To edit vllm recipes locally, point vllm at a checkout with sources.local.yaml; pnpm content:watch re-assembles on file adds/removes while pnpm dev runs.

  • One owner per hf_id: two core/owner sources shipping the same recipe is a build error, unless one declares defers_to: <the other> (a migration aid: its copies of the other's recipes are skipped unparsed, and the rest still has to pass its scope). scope: { tasks: [omni] } / exclude_tasks constrain owner sources.
  • Overlays (src/lib/overlay-merge.js): overlays/<org>/<repo>.yaml merges into another source's recipe, add-only (overwriting a core key is an error) and scoped to the hardware scope.brands grants in sources.yaml (never self-declared): in-scope profile ids / generations (npu) / brand keys under the hardware-keyed maps listed in OVERLAY_RULES, new variants whose supported_hardware is all in scope, in-scope ids appended to an existing variant's supported_hardware, omni.tasks.<key>.hardware_overrides.<hw>, dependencies with an in-scope brand (inserted before core deps unless position: end), and guide_sections: { <name>: md } filling <!-- overlay:<source>:<name> --> markers in the core guide (else appended; unfilled markers are stripped). Overlay keys land after core keys, so an overlay variant's pill renders last. When the schema gains a hardware-keyed field, add it to OVERLAY_RULES. scripts/split-overlay.mjs is the inverse (extract a brand's keys from full recipes) for migrations.
  • Lock bumps are automated by .github/workflows/bump-sources.yml (every 20 min, manual dispatch, or repository_dispatch: recipes-updated from a source repo): sync-sources --update, then build-recipes-api + test-overlay validate the new content, and scripts/lock-changes.mjs writes the commit message for the signed-off bot commit to main that Vercel deploys. A failure leaves the lock on the last good commit and opens one sources-bump-failed issue per failure, not per run: each report carries a fingerprint of the target commits, so the same failing target re-running every 20 minutes adds nothing, a new failing target adds one comment, and an error before any target resolves (e.g. a GitHub API blip) is reported only on its second run in a row. The next successful run closes the issue.
  • Source-repo PR checks: .github/workflows/validate-recipes.yml is a reusable workflow (workflow_call, inputs source/path/renderer-ref) that source repos call on pull requests — vllm's .github/workflows/recipes.yml uses @main. It checks out the PR's recipe dir plus this renderer and runs scripts/validate-source.mjs --source <name> --dir <path>: sync → assemble --strict → schema + JSON API build, the same validation a lock bump runs. Read-only, no secrets, fork-safe. The script writes a temp sources file with an isolated validate channel (a copy of the default channel with that source read from --dir; or, for a source not in sources.yaml yet, --dir alone as a core source) and points the child scripts at it via RECIPES_SOURCES / RECIPES_CHANNEL / RECIPES_PUBLIC_DIR, so .content/latest, public/ and the developer's sources.local.yaml (read from beside the sources file in use, like the lock) are untouched. RECIPES_OVERRIDES=name=dir is also honored by loadOverrides for ad-hoc runs.
  • Developers preview a local checkout of a source with sources.local.yaml (gitignored): overrides: { vllm: ~/vllm/recipes }. RECIPES_SOURCES points the scripts at an alternate sources file (its lock sits beside it).

Data pipeline

.content/<channel>/models/<org>/<repo>.yaml  ──┐   (assembled from sources.yaml)
strategies/*.yaml          ─┼──► src/lib/*.js  ─► Next.js pages + JSON API in public/
kv_store/*.yaml            ─┤       (SSG, no runtime DB)
taxonomy.yaml              ─┘
  • taxonomy.yaml — the controlled vocabulary: hardware profiles (GPUs/node + VRAM), tasks, serving strategy ids, plus the kv_offload block — the composing KV-offload options (Simple, LMCache), each { display_name, label?, description, args[], strategies?, install?, companion? }, read data-driven by command synthesis. strategies is a per-option serving-strategy allowlist (LMCache = single-node only); companion renders as an extra command tab (e.g. lmcache server) while the option is active.
  • strategies/*.yaml — the 9 serving strategies: single_node_{tp,tep,dep}, multi_node_{tp,tep,dep,tp_pp,tp_dp}, pd_cluster. Each defines the vLLM flags and env for that parallelism pattern. multi_node_tp_dp is the hybrid TP-per-node + DP-across-nodes layout (parallelism: tp_dp in the YAML → synthesis emits --tensor-parallel-size <gpus/node> --data-parallel-size <nodes> --data-parallel-size-local 1, worker node N = DP rank N).
  • kv_store/*.yaml — the Mooncake KV-store deployments kv_store_{distributed,centralized}_mooncake (deploy_type: kv_store_lb — router + master + optional store + N vLLM instances, each spanning the Nodes-row node count). They power the KV Offload row, not the Strategy row; loadStrategies() merges them into the same map (looked up by id everywhere) but the JSON API publishes them under public/kv_store/ so /strategies.json stays the serving vocabulary.
  • src/lib/recipes.js, strategies.js, taxonomy.js — filesystem readers that parse YAML once per process.
  • src/lib/command-synthesis.js — the core of the site. Pure functions that take (recipe, variantKey, strategyName, hwProfileId, features, strategies, taxonomy, advancedArgs, nodeCount) and return either a single vllm serve command, a pair of head+worker commands (multi-node), or a prefill/decode pair (pd_cluster). All parallel flag decisions, per-generation hardware overrides, and multi-node mp args (--nnodes/--node-rank/--master-addr for TP, --data-parallel-* for DEP) live here.
  • scripts/build-recipes-api.mjs — writes public/taxonomy.json, public/strategies/*.json, public/kv_store/*.json, and public/<hf_org>/<hf_repo>.json at build time so agents can consume recipes without the website.

Next.js App Router

  • src/app/page.js — homepage (provider grid, alphabetical).
  • src/app/[org]/page.js — org overview, task-grouped sections.
  • src/app/[org]/[repo]/page.js — the recipe detail page. SSG via generateStaticParams over all YAMLs. Renders header + <CommandBuilder> + guide markdown (via react-markdown, not MDX — MDX chokes on <think> and <64K literals in guide text).
  • src/app/[org]/layout.js — wraps every recipe/org page with the left <ModelSidebar>, sorted by HF release date desc.
  • src/components/recipes/CommandBuilder.jsx — the interactive piece. Client component. Holds state for variant, hwId, strategyOverride, nodeCount, features, advanced; syncs all of them to URL query params; renders Hardware / Variant / Strategy / KV Offload / Nodes / Features / Advanced rows + the sticky command block. Calls resolveCommand on every change. The KV Offload row: Off (default) / Simple (taxonomy-driven --kv-transfer-config appended to the current strategy; disabled under pd_cluster) / one merged Mooncake pill. Every option COMPOSES with the serving strategy — the KV layer is orthogonal to parallelism, the Strategy row is always in effect. Mooncake wraps the selected strategy in a kv_store_lb deployment shell (router + master + optional store around N instances, each instance running the serving strategy's exact command — TP/TEP/DEP, single- or multi-node — with the MooncakeStoreConnector args/env appended last-wins; synthesis exposes the pairing as servingStrategy, the JSON API as serving_strategy). Offered on every non-omni recipe by default, needs scalable NVIDIA/AMD hardware (isKvStoreBrandSupported — the transfer engine has no CPU/TPU build; the brand-keyed vllm.install resolves CUDA vs non-CUDA wheel and the kv YAML's min_vllm_version bumps the Install header while active); a recipe opts a GPU out via kv_cache_strategy_hardware: { <kv_store_id>: { <gpu_id>: unsupported } }; URL param kv_offload holds the topology's strategy id. A Mooncake pick reveals a Store Topology row (Distributed embedded vs Centralized standalone store — short label from the kv_store YAML; distributed is the first-click default) and an Instances input (URL kv_instances; default = the topology's default_instances: distributed 1 — a single embedded instance is a complete CPU-offload tier — centralized 2). The router only exists at ≥2 instances; at 1 the client target falls back to VLLM_HOST:8000 and cURL/Bench move to the vLLM tab. The Nodes row keeps meaning nodes PER INSTANCE — relabeled "Nodes / Instance" while Mooncake is active, tooltips multiply totals by the instance count; at >1 each instance shards like multi_node_tp (tabs named Head / Node 1, same as MultiNodeBlock; --master-addr is $HEAD_IP at 1 instance, $VLLM_INSTANCE_N when routed). Under pd_cluster Mooncake COMPOSES instead of replacing: PD keeps Prefill/Decode/Router, each role's Nixl config swaps (last-wins) for the kv_store YAML's pd.<role>.args MultiConnector (Nixl + MooncakeStoreConnector), both roles gain the Mooncake env + config heredoc, and Mooncake Master/(Store) tabs lead the PD block; the Instances row hides (PD pools size themselves). Interaction-expression conventions: the Strategy row description gains an "Each Mooncake instance runs this strategy." suffix while Mooncake is active (composition made explicit, no dimming — the strategy is genuinely in effect); the active KV option's description + Docs link render under the row (Strategy-row style); Store Topology / Instances / the spec-method row use ConfigRow nested (blue left rail = child of the pick above); command-block tabs go through the shared CommandTabs (ARIA tablist, arrow-key roving focus, dimmed step numbers whenever the tabs form a launch sequence — always for Mooncake/kv_store, PD only when composing, companions); cURL/Bench render disabled-with-tooltip (never hidden) on tabs clients don't connect to; disabled Pills wrap in a span so Radix tooltips still explain WHY (native disabled swallows pointer events); the "Verified on " badge yields to the config summary whenever any KV Offload option is active (verification covered the plain serving path); setting KV Offload back to Off resets kv_instances/kv_instance to topology defaults.

Recipe YAML schema (authoritative)

The schema lives in SCHEMA.md — annotated example, field reference, and VRAM formula. It is the single copy: CONTRIBUTING.md and the add-recipe skill link to it, and recipes in every source repo (vllm recipes/, vllm-omni, this repo's omni recipes) follow it. Change it in the same commit as any code that adds or changes a recipe field.

@SCHEMA.md

Machine-readable schema: src/lib/recipe-schema.js builds a JSON Schema (draft-07) from the taxonomy + strategies (hardware / strategy / frontend / task / kv-offload enums never drift); build-recipes-api.mjs publishes it as public/recipe.schema.json and validates every recipe with ajv before rendering, listing all failures at once. Objects are closed (additionalProperties: false), because the renderer silently ignores unknown keys — so adding or renaming a recipe field means updating recipe-schema.js and SCHEMA.md in the same change, or every build fails. Cross-field rules (mode references, hardware semantics, frontends) stay in the build's validate* functions. meta.difficulty is deprecated (nothing reads it): the schema still accepts it, new recipes and the templates omit it, and the vLLM recipes drop it as they move (vllm-project/vllm#61302). Starter template: templates/recipe.yaml — text by default, multimodal features commented in (pnpm test:schema checks recipes + the template against the schema and that misspellings are rejected).

Non-obvious design decisions

  • Hardware is never blocked by VRAM. Multi-node scales VRAM, so command-synthesis.js only uses VRAM as a display hint. Precision constraints (NVFP4/FP4 require Blackwell by default) are the only things that disable a hardware pill; exceptional runtime fallbacks can broaden them with variant-level precision_hardware.
  • Default hardware is always H200 (or B200 for Blackwell-constrained variants). pickDefaultHardware prefers NVIDIA over AMD; loadPreferences deliberately ignores AMD entries from localStorage so first-page-load is always H200.
  • Omni recipes render vllm serve --omni via a dedicated resolveOmniCommand path (no strategy / multi-node / pd). The command builder adds a Task pill row (T2I / I2I / T2V / I2V / TI2V / T2A) above Hardware, with task-specific cURL examples in the popover. Detection: meta.tasks includes omni. The --omni flag is auto-injected, so don't add it to model.base_args.
  • Multi-node uses vLLM mp (multiprocessing) backend, not Ray. For TP/TEP every node runs the same command varying --node-rank; rank>0 adds --headless. For DEP, Node N adds --data-parallel-start-rank = N × gpus_per_node. One command per node, never a repeated one: synthesis renders the head (rank 0) plus one command per follower rank via buildArgs(role, "worker", rank) — followerRanks(nodeCount) = [1 … nodeCount-1] — so the command block shows a tab per node (Head · Node 1 · … · Node N-1) and a 4-node cluster gets three distinct worker commands. Same in the Mooncake block (nodes per instance). The JSON API carries the full list as worker_commands[] / worker_argvs[] (+ worker_docker_*s[], and vllm_worker_*s[] under kv_store_lb); the older singular worker_command/worker_argv stay as rank 1 for compatibility — agents should loop the plural arrays, since replaying rank 1 on every node hangs the rendezvous.
  • Taxonomy hardware specs follow inferencex.semianalysis.com/gpu-specs: B200 SXM is 180 GB/GPU (not the nameplate 192, which is pre-ECC), GB200/GB300 are 4-GPU compute trays (NVL4, the NVL72 rack unit), B300 is 268 GB/GPU.
  • Legacy Markdown under DeepSeek/, Qwen/ etc. is read-only reference. When migrating or updating a recipe, read those to extract flags and env, but write the YAML; don't edit the Markdown.
  • public/ is generated — don't commit it. .gitignore should cover it; if something slips through, leave it untracked.

Contributing a new recipe

A dedicated skill lives at .agents/skills/add-recipe/SKILL.md (symlinked into .claude/skills/) and is auto-loaded when the user asks to add/create/contribute a recipe. It covers the evidence → YAML → UI contract, HF metadata extraction, the authoring loop in the owning repo, local preview through sources.local.yaml, provider-registry updates, and validation; it links SCHEMA.md for fields. Source repos keep only a pointer to it (vllm's model-recipes skill, a recipes/README.md), so authoring guidance is maintained here once.