feat(deps): replace renamed Candle fork with Mold extensions - #717
Merged
Conversation
jamesbrink
force-pushed
the
feat/candle-extension
branch
from
August 5, 2026 05:53
d117bfd to
96caa5c
Compare
Greptile SummaryThe PR replaces Mold-renamed Candle packages with official Candle identities plus a narrow compatibility patch, moves Mold-owned extensions into
Confidence Score: 5/5The PR appears safe to merge based on the available follow-up review scope. No blocking failure remains in the eligible review set.
|
| Filename | Overview |
|---|---|
| Cargo.toml | Adds the Candle extension crate and same-name Candle compatibility patches while preserving a single upstream package identity. |
| crates/mold-candle/Cargo.toml | Defines the publishable mold-ai-candle extension crate against ordinary Candle 0.11 registry dependencies. |
| crates/mold-candle/src/ltx_video/sampling.rs | Introduces the standalone LTX-Video flow-matching scheduler and its timestep, sigma, and sampling operations. |
| crates/mold-candle/src/ltx_video/transformer.rs | Moves the Mold-owned LTX-Video transformer implementation behind the new extension-crate boundary. |
| crates/mold-candle/src/ltx_video/vae.rs | Moves the LTX-Video VAE implementation into the Mold-owned Candle extension crate. |
| crates/mold-inference/Cargo.toml | Replaces renamed Candle dependencies with official package identities and integrates mold-ai-candle. |
| scripts/release/publish-crates.sh | Centralizes resumable crates.io publication in workspace dependency order with exact-version checks and index-visibility waits. |
| scripts/tests/crates-publish-contract.sh | Verifies complete, unique, topologically ordered publication and partial-release recovery. |
| .github/workflows/release.yml | Replaces individually maintained publication steps with the guarded workspace publisher. |
| Cargo.lock | Consolidates the graph on Candle 0.11 package identities and records the new Mold extension crate. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
C[candle-core / candle-nn / candle-transformers 0.11] --> MC[mold-ai-candle]
MC --> MI[mold-ai-inference]
MI --> MS[mold-ai-server]
MS --> MA[mold-ai]
PUB[publish-crates.sh] --> C0[mold-ai-core]
C0 --> DB[mold-ai-db]
DB --> CAT[mold-ai-catalog]
CAT --> MC
MC --> MI
MI --> MS
MS --> MA
Reviews (3): Last reviewed commit: "fix(release): publish complete crate dep..." | Re-trigger Greptile
Member
Author
|
Addressed Greptile’s quantized-FLUX documentation finding in rebased commit 20032d2: both runtime-LoRA and merged/requantized GGUF loads use the Mold-owned |
Bump candle-core-mold / candle-nn-mold / candle-transformers-mold to 0.11.1. The 0.11 API removed the LTX-Video transformer's per-layer skip_layer_mask forward argument in favor of the set_skip_block_list hard skip, so the STG perturbed pass now sets the block list around its forward call — output-equivalent to the old 0/1 mask (identity passthrough) and faster, since skipped blocks are no longer executed. The now-dead create_skip_layer_mask helper is removed. 0.11.1 is the fork release restoring two patches the 0.11.0 upstream sync dropped (DiagonalGaussianDistribution::mode() and high-resolution attention-score bounding); a temporary [patch.crates-io] section points at the local fork checkout until that tag is published, and must be removed once 0.11.1 resolves from crates.io.
codex review caught that the desktop crate has its own Cargo root and lockfile, which still pinned the three candle fork crates at 0.9.13 and does not inherit the workspace's temporary [patch.crates-io]. Mirror the temporary local-fork patch there and refresh its lockfile; both patch sections are removed together once fork v0.11.1 is on crates.io.
jamesbrink
force-pushed
the
feat/candle-extension
branch
from
August 5, 2026 21:13
2409104 to
8660099
Compare
Member
Author
Member
Author
10 tasks
jamesbrink
added a commit
that referenced
this pull request
Aug 6, 2026
The `mold_flash_attn_real` gate is gone (#717), but the configuration it left behind had never been compiled by anything. `--features flash-attn` failed `clippy -D warnings` on an unreachable expression plus a dead static and function: the feature-on arm of `flash_attention` ended in `return`, so the trailing math call and the whole one-shot-warning apparatus were reachable only in a build without the feature. Split the two paths into cfg-selected functions, which also avoids the `needless_return` the obvious alternative reintroduces. Make the fallback genuinely safe. `flash_is_eligible` now rejects head dims FlashAttention v2 cannot serve. FLUX, Flux.2 and LTX-2 are pinned at 128 by their configs, but Z-Image reads `head_dim` off the loaded checkpoint and Mold accepts arbitrary `cv:`/`hf:` exports, so an unsupported value was user-reachable and aborted a generation the math path would have completed. Stop copying q/k/v into the kernel. The FFI wrapper reads outer strides off the layout and requires only a unit stride on the last axis, so the transposed view was always a valid argument and the three full-size buffers per call were never required. Worth 1.4-2.2x on the flash path, pinned bit-exact against the copy by a CUDA-gated test. Add the `flash-attn-check` CI job. It names `mold-ai-inference` directly because `-p mold-ai --all-targets` builds that crate as a dependency lib and never compiles its test module, and it caches the kernel objects and cudaforge's CUTLASS checkout so the cold build stays a once-per-cache cost rather than an hour per run. Measured on an RTX 4090 at BF16 with the new `attention_bench` dev-bin, flash is 6.0-9.2x faster than math on the attention operation itself across FLUX, Flux.2, Z-Image and LTX-2 shapes -- the operation alone, not end-to-end generation. No release artifact enables the feature, so no shipped binary changes behaviour. Refs #598
jamesbrink
added a commit
that referenced
this pull request
Aug 6, 2026
#733) Closes #598. ## The premise changed under the issue #598 was filed against a `mold_flash_attn_real` RUSTFLAGS gate. #717 removed that gate a week later by retiring the renamed `candle-*-mold` packages in favour of a same-name `[patch.crates-io]` branch, so the crates.io `candle-flash-attn` now resolves its `candle-core` through the same patch and shares one `Tensor`. `cargo check -p mold-ai --features flash-attn` already succeeds with no special RUSTFLAGS, and `nvcc` really compiles the FA2 kernels. What #717 did **not** do is make that configuration compile *clean*, and nothing in CI ever built it — which is exactly how the feature rotted the first time. ## What this changes **The cfg asymmetry.** `--features flash-attn` failed `clippy -D warnings` on three counts: an unreachable expression plus a dead static and function. One root cause — the `#[cfg(feature = "flash-attn")]` arm of `flash_attention` ended in `return`, so the trailing math call and the whole one-shot-warning apparatus were live only in a build *without* the feature. The two paths are now cfg-selected functions instead of cfg blocks sharing a body, which also sidesteps the `needless_return` that the obvious alternative reintroduces. **A genuinely safe fallback.** `flash_is_eligible` now also rejects head dims FA2 cannot serve (not a multiple of 8, or above 512). This is not defensive padding: FLUX, Flux.2 and LTX-2 are pinned at 128 by their configs, but Z-Image reads `head_dim` off the loaded checkpoint and Mold will load an arbitrary `cv:`/`hf:` export — so an unsupported value was user-reachable and aborted the generation where math would have completed it. **Three copies per call that were never required.** The FFI wrapper reads outer strides off the layout and requires only a unit stride on the last axis (`candle-flash-attn-0.11.0/src/lib.rs:60-68`, strides forwarded at `:178-190`), so the transposed *view* is already a valid argument. Dropping the copies is worth 1.4–2.2× on the flash path alone and is pinned bit-exact against the old copy by a CUDA-gated test. **A CI gate.** New `flash-attn-check` job. Three details that matter: - It names `mold-ai-inference` directly, because `-p mold-ai --all-targets` builds that crate as a plain dependency lib and never compiles its test module — the command that originally found this bug cannot validate the fix. `dev-bins` is included so `attention_bench` is type-checked too; nothing else compiles the dev-bins. - It caches both the kernel objects and cudaforge's CUTLASS checkout — an unretried `git clone` inside a build script that `rust-cache` does not cover — and pins `CANDLE_FLASH_ATTN_BUILD_DIR` so a cargo metadata-hash change doesn't silently cost another hour. cudaforge decides freshness by content hash rather than mtime, so a restored directory short-circuits the whole 53-kernel build. - It runs on every `main` push and on PRs touching the dispatcher or the feature wiring, on a filter narrower than `gpu`. Running on `main` is not redundant: GitHub evicts caches after 7 idle days, so a rarely-triggered job would be cold every single time. Expect **this PR's own run to be cold** (roughly an hour): there is no cache on `main` yet, and `save-if` restricts saving to `main`, so the first warm run is the post-merge one. That is also the point — the job compiles on this PR, so the gate is exercised before it is trusted. ## Measured, on an RTX 4090 at BF16 Via the new `dev-bins` `attention_bench`, at shapes taken from the engines rather than round numbers: | shape | math (ms) | flash (ms) | speedup | max abs diff | | --- | --- | --- | --- | --- | | FLUX offload 768² | 5.43 | 0.67 | 8.10× | 5.9e-3 | | FLUX offload 1024² | 11.47 | 1.70 | 6.75× | 6.4e-3 | | FLUX offload 1536² | 58.15 | 7.55 | 7.70× | 2.4e-3 | | Flux.2 klein 1024² | 11.23 | 1.63 | 6.88× | 4.9e-3 | | Flux.2 dev 1024² | 21.53 | 3.58 | 6.01× | 3.9e-3 | | Z-Image offload 1024² | 13.64 | 1.88 | 7.27× | 3.9e-3 | | LTX-2 stage 2 default | 114.82 | 12.99 | 8.84× | 2.9e-3 | | LTX-2 stage 2 1024² | 175.18 | 18.99 | 9.23× | 2.4e-3 | Two honest caveats. **This is the attention operation in isolation, not end-to-end generation**, where attention is one term among many — do not read 9× as a 9× faster render. And the diffs are bf16-scale (eps ≈ 7.8e-3), not bug-scale: FA2 keeps an fp32 online-softmax accumulator while the math path reduces in the input dtype, so they agree to bf16 precision by construction, not to `1e-5`. Forcing the old `.contiguous()` copies back for an A/B gave 3.12–6.60× instead of 6.01–9.23×, with byte-identical max-diff values in both runs. ## Deliberately not in this PR - **The feature still ships in nothing.** `release.yml` (all five build jobs), `Dockerfile`, the AUR PKGBUILD and the Nix flake all build without it, so no released binary is affected by any of this. Shipping it is a real decision rather than an oversight: compiling the feature flips `default_backend()` to `Flash`, which would silently change every CUDA user's output for a given seed and break the cross-backend seed reproducibility Mold currently guarantees. If we want to ship it, the clean shape is to ship the kernel but leave the default on `math` — that decouples "available" from "on", and it belongs in its own PR. - **The dispatcher's reach is narrower than the issue assumed.** Only four call sites use it: FLUX's offload and GGUF-bypass transformers, all of Flux.2, and Z-Image's offload transformer. Dense BF16 FLUX and Z-Image go through upstream Candle's own attention, so `--features flash-attn` does nothing for the most common 24 GB FLUX configuration. Closing that means enabling `candle-transformers/flash-attn`, which needs `candle-flash-attn` added to `[patch.crates-io]` first — the fork declares it as a path dep, so a bare feature flip would pull a second instance of the crate. Filed separately. - **LTX-2 keeps its own attention.** It deliberately computes in F32 and takes an additive mask FA2 cannot accept, so routing it through the dispatcher is a numerics change needing video-level validation, not a wiring change. The LTX-2 rows above exist to size that follow-up. ## Peer review `codex review --base origin/main` raised exactly one finding, which I am **declining** — with the reasoning, since being able to say why a finding does not apply is the point of running it: > **[P1] Avoid `is_multiple_of` beyond the declared MSRV** (`attention.rs:327`) — integer `is_multiple_of` stabilized in Rust 1.87, but `Cargo.toml` declares `rust-version = "1.85"`. The API version is correct; the conclusion is not. Three facts against it: 1. **It cannot be "introduced" here.** `origin/main` already has **27 occurrences across 11 files** in `crates/` (`git grep -c is_multiple_of origin/main -- 'crates/**/*.rs'`). The effective MSRV has been ≥1.87 for a long time; this line changes nothing about that. 2. **The suggested fix fails a required gate.** Writing `head_dim % 8 == 0` makes `cargo clippy --workspace --all-targets -- -D warnings` fail with `clippy::manual_is_multiple_of`, which `ci.yml` enforces. Codex's remedy would turn a green PR red, and silencing it with an `#[allow]` would make this one call site diverge from the other 27. 3. **Nothing verifies the declaration.** No workflow pins or tests a toolchain (`grep -rn 'rust-version\|MSRV\|1\.85' .github/workflows/` is empty), so the 1.85 in `Cargo.toml` is documentation, and stale documentation at that. The real defect Codex is circling is that **`rust-version = "1.85"` is wrong** and has been for a while. That is pre-existing, affects `Cargo.toml`, `CLAUDE.md` and the AUR PKGBUILD, and needs the true minimum established by actually building on a candidate toolchain — so it gets its own issue rather than a guess bolted onto this one. ## Verification Local, all green: `cargo fmt --all --check`; `cargo clippy --workspace --all-targets -D warnings`; `cargo test --workspace`; `cargo check -p mold-ai --features preview,discord,expand,tui,webp,mp4`; `cargo clippy -p mold-ai-inference --features flash-attn --all-targets -D warnings` (the new gate, which fails on `main`); attention tests in *both* feature configurations, including the CUDA-gated layout test on a real 4090; `actionlint` on the workflow; web (1165 tests), desktop (2933 tests, `vue-tsc`), studio (401 tests), website `verify`/`build`, and every affected `fmt:check`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
candle-*-moldpackages with official Candle package identities and a narrow same-name compatibility patchmold-ai-candlefor Mold-owned LTX-Video models, Stable Diffusion component policy, in-memory GGUF builders, and efficient quantization helperscandle-flash-attndirectly without privateRUSTFLAGSmold-ai-candleand previously omittedmold-ai-schedulerWhy
Renaming the fork created a second
candle_core::Tensoruniverse, broke ecosystem interop, and forced Mold to republish three framework crates for application-owned model code. The new boundary keeps one Candle type identity while letting Mold extend public APIs in a normal workspace crate. Only backend/private changes remain in the compatibility branch.The v0.21.0 crates.io release also exposed a publication-DAG defect:
mold-ai-schedulerwas omitted, so publication stopped atmold-ai-serverafter several immutable versions had landed. Addingmold-ai-candlewould have created the same failure beforemold-ai-inference. A single guarded publisher now owns the complete topological order and supports safe retries.crates.io behavior
[patch.crates-io]remains a workspace/source-build concern and is not inherited by published dependentsmold-ai-candledeclares ordinary Candle0.11registry dependencies and verifies against stock crates.io Candlescripts/release/publish-crates.shpublishes all ten workspace crates in dependency order with--lockedUpstream work
Validation
cargo fmt --all -- --checkcargo check --workspacecargo clippy --workspace --all-targets -- -D warningscargo test --workspacecargo check -p mold-ai --features metal,preview,expand,tui,webp,mp4,mdnscargo check --manifest-path desktop/src-tauri/Cargo.tomlcargo test -p mold-ai-candle(6 passed)cargo package -p mold-ai-candle --allow-dirtyagainst unpatched crates.io Candle 0.11bash scripts/tests/crates-publish-contract.shbash scripts/tests/release-sync-pr.shbash scripts/tests/ci-routing-contract.shbash scripts/tests/desktop-candle-lock-sync.shnix flake checknix build .#mold-desktop --no-linkcandle-core,candle-nn, orcandle-transformerssourceCUDA execution remains hardware validation; CI directly typechecks the CUDA and real Flash Attention feature path.
Closes #704
Addresses #598