feat: Add DGX Spark cluster support, TP awareness, use-case/availability filters, and live benchmarking#181
Closed
michaeljabbour wants to merge 325 commits into
Closed
Conversation
Add MlxProvider implementing ModelProvider for Apple MLX framework. Detects MLX availability via server probe and mlx_lm Python package check. Scans ~/.cache/huggingface/hub/ for installed mlx-community models. Supports pulling via huggingface-cli with Python fallback. Wire into TUI for multi-provider installed-model detection: models show as installed if found in either Ollama or MLX. Provider-aware download selects MLX for MLX-runtime models, Ollama otherwise. - Add MlxProvider with is_available(), installed_models(), start_pull() - Add hf_name_to_mlx_candidates() with ~27 model mappings + heuristic - Add is_model_installed_mlx() and mlx_pull_tag() helpers - Add mlx_available, mlx_installed, mlx fields to App - Update start_download() to choose provider based on ModelFit::runtime - Update refresh_installed() to query both providers - Add 3 new tests for MLX name matching and cache detection Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of trying a Python fallback when huggingface-cli fails, we now provide actionable feedback telling users to install huggingface-cli using uv tool install 'huggingface_hub[cli']'. This avoids issues with global Python package installation being a logistical minefield.
Display MLX vs llama.cpp runtime distinction across all output modes so users can see and act on the inference runtime selection. TUI changes: - Add MLX status in system bar alongside Ollama - Add Runtime line in detail view with estimated tok/s - Show per-provider installed status (✓ Ollama ✓ MLX) - Enable d/i/r keybindings when any provider is available CLI changes: - Add Runtime column to CLI table output - Add runtime info to display_model_detail() - Add "runtime" and "runtime_label" fields to JSON output - Add --runtime filter to recommend subcommand (mlx, llamacpp, any) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
APU integrated GPUs like 'AMD Radeon(TM) Graphics' were matching the generic 'radeon' fallback and getting 8 GB estimated VRAM, when they typically have only 0.5 GB dedicated VRAM. This caused the iGPU to be ranked above discrete GPUs, affecting model recommendations. Now detect APU iGPUs by checking for 'radeon/amd' + 'graphics/igpu' without a discrete model number (RX/HD/R5/R7/R9) and return 0.5 GB. Fixes AlexsJones#25
…m-estimation fix: correctly estimate VRAM for APU integrated GPUs
Signed-off-by: Alex <alexsimonjones@gmail.com>
Signed-off-by: Alex <alexsimonjones@gmail.com>
…ches--main--components--llmfit chore(main): release llmfit 0.3.5
Signed-off-by: Alex <alexsimonjones@gmail.com>
feat: release plumbing
…ches--main--components--llmfit chore(main): release llmfit 0.3.6
Signed-off-by: Alex <alexsimonjones@gmail.com>
feat: plumbing 2
…ches--main--components--llmfit chore(main): release 0.3.7
Signed-off-by: Alex <alexsimonjones@gmail.com>
Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxes53235@gmail.com> Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
feat: MLX-native inference support for Apple Silicon
The name 'huggingface-cli' is deprecated. Their CLI is now called 'hf': https://huggingface.co/docs/huggingface_hub/en/guides/cli
fix: invoke hf instead of huggingface-cli
Restructure single-crate project into Cargo workspace: - llmfit-core: core library (hardware detection, model fitting, providers) - llmfit-tui: CLI/TUI binary (unchanged user experience) - llmfit-desktop: macOS desktop app via Tauri 2 The workspace split enables the desktop app to reuse core logic while keeping the CLI/TUI as the default build target. Moved SortColumn to core crate for shared use across frontends. Desktop app features: - System specs display (RAM, CPU, GPU) - Model compatibility table with fit scoring - Dark theme UI using project icon from assets/icon.svg - Tauri 2 with minimal permissions No changes to data files — moved as-is via git mv. Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxes53235@gmail.com> Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
- Remove llama title/subtitle header from desktop app - Show total + available RAM separately - Render all detected GPUs with VRAM, backend, and count - Show unified memory indicator for Apple Silicon - Responsive grid layout for system spec cards Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
…t/workspace-and-desktop-v2 feat: workspace restructure + Tauri desktop app
Adds build-desktop job that builds Tauri desktop app for both aarch64-apple-darwin and x86_64-apple-darwin targets. DMGs are uploaded alongside CLI tarballs in GitHub Releases. Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
…t/workspace-and-desktop-v2 ci: build macOS desktop app (.dmg) in release workflow
Document Docker usage in README.md
Signed-off-by: AlexsJones <alexsimonjones@gmail.com>
Signed-off-by: AlexsJones <alexsimonjones@gmail.com>
AlexsJones
added
documentation
Signed-off-by: AlexsJones <alexsimonjones@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…port feat: add AWQ/GPTQ support with vLLM inference runtime
Signed-off-by: AlexsJones <alexsimonjones@gmail.com>
Signed-off-by: AlexsJones <alexsimonjones@gmail.com>
Add cluster-aware hardware detection and model scoring for DGX Spark clusters (2-3+ nodes with GB10 Blackwell GPUs connected via QSFP). New features: - `llmfit cluster init` — interactive setup with Ray API auto-discovery - `llmfit cluster status` — show cluster config and Ray connectivity - `llmfit cluster remove` — remove saved cluster configuration - `--cluster` / `--no-cluster` flags to control cluster mode - Auto-loads saved config from ~/.config/llmfit/cluster.toml - New VLlm inference runtime and TensorParallel run mode - Cluster-aware TUI system bar showing aggregated resources - JSON output includes cluster metadata Cluster specs per DGX Spark node: - NVIDIA GB10 Grace Blackwell (1 PFLOP FP4 with sparsity) - 128 GB LPDDR5x unified memory (~85 GB usable for models) - 20-core ARM CPU (10× X925 + 10× A725) - QSFP 200 Gb/s interconnect for NCCL tensor parallelism Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three changes in this commit: 1. Remove PII: Changed default head node IP from user's actual network address to generic 10.0.0.1 placeholder. 2. Use case filter: Added cycle filter [u] in TUI filter bar for General/Coding/Reasoning/Chat/Multimodal/Embedding. Shortened search bar to accommodate. 3. TP compatibility: Models now report valid tensor parallelism sizes based on attention head counts. In cluster mode, llmfit automatically selects the best TP degree that the model supports (e.g., TP=2 for most models with 64 heads on a 3-node cluster, since 64 is not divisible by 3). Notes explain free GPUs and suggest PP fallback. Includes head count inference for Qwen, Llama, DeepSeek, Mistral, Gemma, Phi, and MiniMax families. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Sort by year: new "Year" sort option (press [s] to cycle) groups models by release year, newest first. Date column now shows year instead of YYYY-MM to save space. - TP column: new "TP" column in the table shows valid tensor parallelism sizes for each model (e.g., "2,4,8"). In cluster mode, green = supports full cluster TP, yellow = supports TP=2 only. - TP filter: press [x] to cycle through All → TP=2 → TP=3 → TP=4. Filters to only show models compatible with that TP degree. Useful for finding which models can actually use all your cluster GPUs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New `bench` subcommand measures actual inference performance against live providers: llmfit bench # auto-detect provider and model llmfit bench --model qwen3-32b # specific model llmfit bench --provider vllm # target vLLM cluster llmfit bench --runs 5 --json # multiple runs, JSON output Supports three backends: - Ollama: uses /api/generate with native timing (eval_duration, prompt_eval_duration) for precise TPS and TTFT - vLLM: uses /v1/chat/completions (OpenAI-compatible) with wall-clock timing and usage token counts - MLX: same OpenAI-compatible protocol Auto-detection priority: cluster config (vLLM) → Ollama → MLX. Includes warmup run, configurable number of benchmark runs, and statistical summary (avg/min/max TPS, avg TTFT, avg latency). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `llmfit bench --all` discovers and benchmarks every model across all reachable providers (Ollama, vLLM, MLX). Shows per-model results plus a comparison table at the end. - `llmfit bench --url http://host:port` overrides the provider URL, useful when cluster config IP doesn't match or for ad-hoc testing. - Auto-detects models from vLLM /v1/models endpoint when --url is provided, no need to also specify --model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive README update covering all llmfit-spark features: - Renamed project header to llmfit-spark with upstream attribution - DGX Spark cluster support: init, status, remove, config format - Tensor parallelism compatibility: TP column, filter, head count math - Benchmarking: bench command, --all, --url, metrics explanation - Updated keybindings table with u (use case), x (TP filter) - Updated project structure to reflect workspace layout - Added vLLM and TensorParallel to run modes and speed constants - Updated dependencies table Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix non-exhaustive RunMode::TensorParallel match (fit.rs, serve_api.rs, desktop) - Fix keybinding collision: clear_compare_mark moved to 'X' - Eliminate double ClusterSpecs::load() in detect_specs - Replace wrapping_add with checked_add in increment_ip - Add warmup error handling, IP validation, node count feedback - Make vLLM port configurable via VLLM_PORT env var - Document DGX Spark constants and TTFT estimation - Add 38 unit tests (models TP, cluster, bench) - Rebase README: remove fork branding, preserve sympozium link - Fix contradictory streaming comment in bench.rs
michaeljabbour
force-pushed
the
feature/all-spark-features
branch
from
2de3e05 to
4152ede
Compare
This was referenced Mar 22, 2026
Contributor
Author
Closing this PR — split into 3 focused PRsThanks for the thorough review @AlexsJones! We noticed several of our original features have been independently implemented in main since this PR was opened:
Thank you for integrating these — the implementations look great. We've rebased the remaining unique features onto current
All original review items addressed across the 3 PRs:
Closing this PR in favor of the focused ones above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds several features to llmfit from the llmfit-spark fork, all applied additively on top of current upstream/main. The diff is +2,408 / -75 across 14 files — purely additive.
Features
1. DGX Spark Cluster Support (
llmfit cluster)clustersubcommand withinit,status,removeactionsllmfit-core/src/cluster.rs(542 lines):ClusterSpecsstruct, Ray API discovery, TOML config persistence--cluster/--no-clusterglobal flags to override hardware detection with cluster resources2. Tensor Parallelism Awareness
supports_tp(tp_size)andvalid_tp_sizes()methods onLlmModelnum_attention_heads/num_key_value_headsfields added toLlmModel(optional, heuristic inference fallback)xkey3. Use-Case Filter + Availability Filter
ukey): All → General → Coding → Reasoning → Chat → Multi → Embedakey): All → GGUF Avail → Installed (checks llmfit-core GGUF source metadata)apply_filters()4. Year Sort Column
SortColumn::ReleaseYear— sort by release year (newest first), secondary sort by scoreYearcolumn in TUI tableskey cycles now includes Year in the rotation5. Live Inference Benchmarking (
llmfit bench)llmfit-core/src/bench.rs(610 lines):BenchTargetenum,BenchResult, provider-specific benchmark functions--provider auto)--allflag benchmarks every discovered model across all running providers--urlflag overrides the provider endpoint--json) for tool/agent integrationdiscover_all_targets()queries running Ollama/vLLM/MLX instancesFiles Changed
llmfit-core/src/cluster.rsllmfit-core/src/bench.rsllmfit-core/src/models.rssupports_tp(),valid_tp_sizes()llmfit-core/src/fit.rsllmfit-core/src/hardware.rscluster_mode,cluster_node_count,total_gpu_vram_gbonSystemSpecsllmfit-core/src/lib.rsllmfit-core/Cargo.tomltomldependency for cluster config persistencellmfit-tui/src/main.rscluster,benchsubcommands;--cluster/--no-clusterflagsllmfit-tui/src/tui_app.rstp_filterfieldllmfit-tui/src/tui_ui.rsllmfit-tui/src/tui_events.rsxkey (TP filter),akey (availability filter)llmfit-tui/src/display.rsREADME.mdCargo.lockNotes
ClusterSpecsconfig is stored at~/.config/llmfit/cluster.tomlllmfit benchexits cleanly if none found