Add LoRA loading support for MiniMax-H3 by apolinario · Pull Request #14408 · huggingface/diffusers

apolinario · 2026-08-06T16:33:14Z

Adds MiniMaxH3LoraLoaderMixin so MiniMaxH3ModularPipeline loads LoRAs through the standard load_lora_weights path, plus a load time converter for the non diffusers formats in circulation.

What loads

PEFT native diffusers format.
ai-toolkit H3 LoRAs (diffusion_model. prefixed, fused projections).
larryvrh/MiniMax-H3-Turbo-Lora, the 4 step turbo LoRA, in its original unprefixed layout. num_inference_steps=5 matches upstream --steps 4; the two scheduler design already covers its dual schedule sampler, so no custom sampler is needed.
InstantX/MiniMax-H3-Turbo-Lora-Diffusers, the pre converted mirror, without the manual network_alphas workaround its card currently requires.

Conversion

The converter renames onto MiniMaxH3Transformer3DModel, splits the fused QKV LoRA (shared lora_A, row split lora_B), and swaps the SwiGLU fc1 halves from [gate; value] to the converted base's [value; gate]. Verified against ComfyUI's own calculate_weight and ai-toolkit's own merge_out on real base weights: converted factor matrices are bitwise identical after the layout mapping, and effective weights agree to float accumulation noise (4.4e-7 abs over 3.2e9 elements). The output is also key for key identical to InstantX's independent conversion.

Because these files carry mixed ranks (64 for attention/FFN, 16 for AdaLN) and no alpha keys, the loader synthesizes alpha == rank metadata whenever a state dict brings no alpha information of its own, per component, so every module applies at the intended scale 1.0. Without it the global alpha default mis-scales such adapters (rank 64 modules at 0.25x here); that underlying issue in get_peft_kwargs is general and is fixed separately in #14409.

Both transformer partitions are supported and routed by prefix (transformer, transformer_ref), including pipelines loaded with workflow="ref2va" where only transformer_ref exists.

Notes

ai-toolkit defaults to training against the pruned checkpoint, whose AdaLN input width differs from the release weights; such LoRAs fail with a module named shape error in any framework, ours included. Documented on the docs page.
fuse_lora() into bfloat16 is lossy for adapters this small relative to the base weights; the docs note to prefer the default unfused path.
14 new tests in the modular H3 test module (three formats, mixed rank scaling asserted at PEFT level, both partitions, ref2va routing, existing adapter metadata preserved).

bghira · 2026-08-06T17:32:09Z

this could be entirely avoided if #14410 would be addressed by the team for future model releases.

all of the conversions here are simply counter-productive;

swapping the gates is unexplained, simply looks like "just because we could"
splitting qkv projections increases cuda launch overhead substantially and slows down training & inference quite a lot

sayakpaul

Thanks! Have we tried any of the LoRAs mentioned and inferred with them?

sayakpaul · 2026-08-07T05:26:02Z


+## LoRA
+
+`pipe.load_lora_weights` accepts the diffusers/PEFT format and the two formats real MiniMax-H3 LoRAs actually ship in — [ostris/ai-toolkit](https://github.com/ostris/ai-toolkit)'s `diffusion_model.`-prefixed output and unprefixed original-checkpoint keys — converting the latter two onto the transformer's module names, splitting the fused `attn.qkv_proj` into `to_q` / `to_k` / `to_v` and swapping the two halves of the fused SwiGLU projection.


We don't accept the PEFT format though (which includes adapter_config.json.

sayakpaul · 2026-08-07T05:26:20Z

+pipe.load_components(dtype=torch.bfloat16)
+pipe.to("cuda")
+
+pipe.load_lora_weights("some-user/some-minimax-h3-lora", weight_name="lora.safetensors", adapter_name="style")


Specifying weight_name isn't needed.

sayakpaul · 2026-08-07T05:27:16Z

 )
 ```

+## LoRA


I don't think we have to mention it like this (following other pipelines)? A LoRA badge should be enough?

sayakpaul · 2026-08-07T05:30:36Z

+        if metadata is None and not any(k.endswith(".alpha") for k in state_dict):
+            metadata = {}
+            for prefix in (cls.transformer_name, cls.transformer_ref_name):
+                component_state_dict = {
+                    k.removeprefix(f"{prefix}."): v for k, v in state_dict.items() if k.startswith(f"{prefix}.")
+                }
+                # `^` anchors each pattern to a full module name, as `load_lora_adapter` does for the ranks it derives.
+                rank = {f"^{k}": v.shape[1] for k, v in component_state_dict.items() if "lora_B" in k and v.ndim > 1}
+                if not rank:
+                    continue
+                lora_config_kwargs = get_peft_kwargs(
+                    rank, network_alpha_dict=None, peft_state_dict=component_state_dict, is_unet=False
+                )
+                # The same fix-up `PeftAdapterMixin.load_lora_adapter` applies to SAI control LoRAs.
+                lora_config_kwargs["lora_alpha"] = lora_config_kwargs["r"]
+                lora_config_kwargs["alpha_pattern"] = lora_config_kwargs["rank_pattern"]
+                metadata.update(_pack_dict_with_prefix(lora_config_kwargs, prefix))


We should be able to infer all of this here:

diffusers/src/diffusers/loaders/peft.py

Line 80 in 9c6a68c

def load_lora_adapter(

Does it not suffice?

Nevermind. It's needed to do the alpha-related shenanigans.

sayakpaul · 2026-08-07T05:32:04Z

+
+    _lora_loadable_modules = ["transformer", "transformer_ref"]
+    transformer_name = TRANSFORMER_NAME
+    transformer_ref_name = MINIMAX_H3_TRANSFORMER_REF_NAME


Do we have a LoRA with this? If not, do we want to remove it?

sayakpaul · 2026-08-07T05:32:59Z

+            )
+
+    @classmethod
+    def load_lora_into_transformer(


Could use a "Copied from ..." here?

sayakpaul · 2026-08-07T05:36:58Z

+if is_peft_available():
+    from peft.utils import get_peft_model_state_dict
+


We can remove this for now. Because LoRA tests should generally be shipped to all the modular pipelines that support LoRAs.

sayakpaul · 2026-08-07T05:44:24Z

+        # and never re-derives it, so a mixed-rank adapter — which the public turbo LoRA is, rank 64 for attention
+        # and FFN against rank 16 for the AdaLN projections — has one of its two rank groups silently scaled by


Is it rank 64 for attention and FFN and rank 16 for AdaLN? If so, I would clarify this comment.

sayakpaul · 2026-08-07T05:46:25Z

+        if is_non_diffusers_format:
+            state_dict = _convert_non_diffusers_minimax_h3_lora_to_diffusers(state_dict)
+
+        # Every published MiniMax-H3 LoRA is alpha-less and applies as `W + lora_B @ lora_A`, i.e. at an effective


It's also interesting to see how pipeline-specific this is. I don't think there's a way to faithfully derive this info just from a canonical state dict. @BenjaminBossan thoughts?

I don't see how this info could possibly derived from the state_dict. The only way to know this is to already know ahead of time that the checkpoint was trained with r == alpha (or that alpha was multiplied into the weights).

If it helps, we could think about allowing lora_alpha=None in PEFT, in which case we assume that it must be equal to r (it's not quite as easy, e.g. what to do about rsLoRA?). That way, there would no longer be the need to set lora_alpha=r everywhere.

Add LoRA support for MiniMax-H3

288276d

yiyixuxu requested a review from sayakpaul August 6, 2026 16:35

github-actions Bot added the size/L PR with diff > 200 LOC label Aug 6, 2026

apolinario changed the title ~~Add LoRA support for MiniMax-H3~~ Add LoRA loading support for MiniMax-H3 Aug 6, 2026

sayakpaul reviewed Aug 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add LoRA loading support for MiniMax-H3 - #14408

Add LoRA loading support for MiniMax-H3#14408
apolinario wants to merge 1 commit into
mainfrom
minimax-h3-lora

apolinario commented Aug 6, 2026 •

edited

Loading

Uh oh!

bghira commented Aug 6, 2026

Uh oh!

sayakpaul left a comment

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

sayakpaul Aug 7, 2026

Uh oh!

BenjaminBossan Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		## LoRA

		`pipe.load_lora_weights` accepts the diffusers/PEFT format and the two formats real MiniMax-H3 LoRAs actually ship in — [ostris/ai-toolkit](https://github.com/ostris/ai-toolkit)'s `diffusion_model.`-prefixed output and unprefixed original-checkpoint keys — converting the latter two onto the transformer's module names, splitting the fused `attn.qkv_proj` into `to_q` / `to_k` / `to_v` and swapping the two halves of the fused SwiGLU projection.

               )
               ```
+              ## LoRA

		if is_peft_available():
		from peft.utils import get_peft_model_state_dict

		# and never re-derives it, so a mixed-rank adapter — which the public turbo LoRA is, rank 64 for attention
		# and FFN against rank 16 for the AdaLN projections — has one of its two rank groups silently scaled by

Uh oh!

Conversation

apolinario commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What loads

Conversion

Notes

Uh oh!

bghira commented Aug 6, 2026

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

apolinario commented Aug 6, 2026 •

edited

Loading