Merge branch 'main' into fix-modular-required-default-14388

Author: Sravan1011

.ai/models.md

@@ -75,6 +75,27 @@ What you pass as `attn_mask=` to `dispatch_attention_fn` determines which backen
 - **Other mask types (structural, BlockMask, etc.)** — if the model requires a different mask pattern, figure out how to support as many backends as possible (e.g. use `window_size` kwarg for sliding window on flash, `BlockMask` for Flex) and document which backends are supported for that model.
 - **Don't declare `attention_mask` (or `encoder_hidden_states_mask`) in the forward signature if you ignore it.** "For API stability with other transformers" is not a reason; readers assume a declared param is honored, and downstream pipelines will pass padding masks that silently get dropped. Some existing models in the repo carry unused mask params for historical reasons — e.g. `QwenDoubleStreamAttnProcessor2_0.__call__` declares `encoder_hidden_states_mask` but never reads it (the joint mask is routed through `attention_mask` instead), and the block-level forward in `transformer_qwenimage.py` declares it but always receives `None`. This is a legacy behavior and should not be replicated in new models.
 
+### Grouped-query attention
+
+Fewer key/value heads than query heads can be spelled two ways. Either pass `enable_gqa=True` to `dispatch_attention_fn` and let the backend broadcast (`transformer_cosmos3.py`), or repeat the key/value heads in the processor after RoPE and pass no flag (`transformer_krea2.py`):
+
+```python
+num_key_value_groups = attn.num_heads // attn.num_kv_heads
+if num_key_value_groups > 1:
+    key = key.repeat_interleave(num_key_value_groups, dim=2)
+    value = value.repeat_interleave(num_key_value_groups, dim=2)
+```
+
+`dim=2` because tensors are `(batch_size, seq_len, num_heads, head_dim)` here. Must be `repeat_interleave`, not `repeat` — the groups are contiguous, and `repeat` gives a silently wrong pairing no shape check catches.
+
+Both compute the same thing, so weigh the two on compatibility and performance and recommend whichever fits the model better.
+
+- **Compatibility.** Most backends do not implement `enable_gqa` yet — flash, FA3, sage, cuDNN and the hub kernels raise on it, as does the context-parallel path. Grep `enable_gqa` in `attention_dispatch.py` for the current list rather than trusting this one; it changes as support lands. The flag limits the model to whichever backends still accept it, while repeating works on all of them.
+
+- **Performance.** Depends on whether the model passes a mask. With a mask, no fused kernel takes a mask *and* mismatched head counts, so SDPA falls back to math and materializes the full `[batch_size, num_heads, seq_len_q, seq_len_kv]` score matrix — no error, no warning, only memory. Without a mask, flash broadcasts inside the kernel and the flag saves the key/value copy. Both effects scale with sequence length and head count, so measure at the model's real shape; `torch.backends.cuda.can_use_flash_attention(params, debug=True)` and `can_use_efficient_attention` print why a kernel was rejected, which is the fastest way to see which one you actually got.
+
+- **Recommendation.** Repeat by default — it is portable and never pathological. Reach for `enable_gqa=True` only when the model never passes a mask *and* the measured saving justifies the narrower backend support. For scale: on Krea 2 at 1024×1024, masked, the flag cost 9.02 GiB and 26.7 ms per call against 0.16 GiB and 4.1 ms repeated; unmasked at the same shape it saved 0.11 GiB and 0.1 ms. `transformer_cosmos3.py` is the in-repo case where it is defensible — causal, never masked.
+
 ## Model class attributes
 
 Each `ModelMixin` subclass can declare class-level attributes that configure optimization features. Each attribute corresponds to a user-facing API — the attribute controls how that feature behaves for the model. When adding a new transformer, set all that apply — skim `transformer_flux.py`, `transformer_wan.py`, `transformer_qwenimage.py` for examples.

.github/workflows/nightly_tests.yml

@@ -342,19 +342,19 @@ jobs:
       matrix:
         config:
           - backend: "bitsandbytes"
-            test_location: "bnb"
+            marker: "bitsandbytes"
             additional_deps: ["peft"]
           - backend: "gguf"
-            test_location: "gguf"
+            marker: "gguf"
             additional_deps: ["peft", "kernels"]
           - backend: "torchao"
-            test_location: "torchao"
-            additional_deps: []
+            marker: "torchao"
+            additional_deps: ["mslk"]
           - backend: "optimum_quanto"
-            test_location: "quanto"
+            marker: "quanto"
             additional_deps: []
           - backend: "nvidia_modelopt"
-            test_location: "modelopt"
+            marker: "modelopt"
             additional_deps: []
     runs-on:
       group: aws-g6e-xlarge-plus
@@ -389,9 +389,12 @@ jobs:
           BIG_GPU_MEMORY: 40
         run: |
           pytest -n 1 --max-worker-restart=0 --dist=loadfile \
+            -m "${{ matrix.config.marker }}" \
             --make-reports=tests_${{ matrix.config.backend }}_torch_cuda \
             --report-log=tests_${{ matrix.config.backend }}_torch_cuda.log \
-            tests/quantization/${{ matrix.config.test_location }}
+            tests/models \
+            tests/quantization \
+            tests/pipelines/testing_utils/quantization.py
       - name: Failure short reports
         if: ${{ failure() }}
         run: |
@@ -439,9 +442,10 @@ jobs:
           BIG_GPU_MEMORY: 40
         run: |
           pytest -n 1 --max-worker-restart=0 --dist=loadfile \
+            -k "TestPipelineQuantization" \
             --make-reports=tests_pipeline_level_quant_torch_cuda \
             --report-log=tests_pipeline_level_quant_torch_cuda.log \
-            tests/quantization/test_pipeline_level_quantization.py
+            tests/pipelines/testing_utils/quantization.py
       - name: Failure short reports
         if: ${{ failure() }}
         run: |

.github/workflows/pr_tests.yml

@@ -114,6 +114,9 @@ jobs:
       run:
         shell: bash
 
+    env:
+      HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }}
+
     steps:
     - name: Checkout diffusers
       uses: actions/checkout@v6
@@ -242,6 +245,9 @@ jobs:
       run:
         shell: bash
 
+    env:
+      HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }}
+
     steps:
     - name: Checkout diffusers
       uses: actions/checkout@v6

docs/source/en/_toctree.yml

@@ -401,6 +401,8 @@
         title: SD3Transformer2DModel
       - local: api/models/skyreels_v2_transformer_3d
         title: SkyReelsV2Transformer3DModel
+      - local: api/models/stable_audio_3_transformer
+        title: StableAudio3DiTModel
       - local: api/models/stable_audio_transformer
         title: StableAudioDiTModel
       - local: api/models/transformer2d
@@ -477,6 +479,8 @@
         title: AutoencoderKLWan
       - local: api/models/autoencoder_rae
         title: AutoencoderRAE
+      - local: api/models/autoencoder_same
+        title: AutoencoderSAME
       - local: api/models/consistency_decoder_vae
         title: ConsistencyDecoderVAE
       - local: api/models/ltx2_diffusion_decoder
@@ -503,6 +507,8 @@
         title: LongCat-AudioDiT
       - local: api/pipelines/stable_audio
         title: Stable Audio
+      - local: api/pipelines/stable_audio_3
+        title: Stable Audio 3
       title: Audio
     - sections:
       - local: api/pipelines/anima

docs/source/en/api/models/autoencoder_same.md

@@ -0,0 +1,42 @@
+<!--Copyright 2025 Stability AI and The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# AutoencoderSAME
+
+The **SAME** (Semantically-Aligned Music Encoder) autoencoder is used by [Stable Audio 3](https://stability.ai/news/stable-audio-3)
+to compress stereo audio waveforms into a compact latent sequence and reconstruct them.
+
+The encoder stacks [`SAMETransformerResamplingBlock`] modules, each of which groups a fixed number of audio
+patch frames and produces one learnable output token via a differential transformer. The decoder inverts this
+process, expanding each latent token back to a patch of audio frames.
+
+A soft-norm bottleneck (`SoftNormBottleneck`) normalises latents before and after the diffusion model,
+providing stable training dynamics.
+
+## AutoencoderSAME
+
+[[autodoc]] AutoencoderSAME
+	- all
+	- encode
+	- decode
+
+## SAMETransformerResamplingBlock
+
+[[autodoc]] models.autoencoders.autoencoder_same.SAMETransformerResamplingBlock
+
+## AutoencoderSAMEOutput
+
+[[autodoc]] models.autoencoders.autoencoder_same.AutoencoderSAMEOutput
+
+## AutoencoderSAMEDecoderOutput
+
+[[autodoc]] models.autoencoders.autoencoder_same.AutoencoderSAMEDecoderOutput

docs/source/en/api/models/stable_audio_3_transformer.md

@@ -0,0 +1,39 @@
+<!--Copyright 2025 Stability AI and The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# StableAudio3DiTModel
+
+A rectified-flow velocity-prediction Diffusion Transformer (DiT) for audio generation, used in
+[Stable Audio 3](https://stability.ai/news/stable-audio-3).
+
+Each [`StableAudio3DiTBlock`] performs:
+
+1. **Self-attention** — differential multi-head attention with rotary position embeddings (RoPE).
+2. **Cross-attention** — attends to the token sequence from the T5Gemma text encoder.
+3. **Feed-forward** — SwiGLU projection.
+
+The model is conditioned on a **timestep** (exponential Fourier features → linear projection) and a **global
+conditioning vector** (duration embedding from [`StableAudio3DurationEmbedder`]).
+
+## StableAudio3DiTModel
+
+[[autodoc]] StableAudio3DiTModel
+	- all
+	- forward
+
+## StableAudio3DiTBlock
+
+[[autodoc]] models.transformers.transformer_stable_audio3.StableAudio3DiTBlock
+
+## StableAudio3DiTModelOutput
+
+[[autodoc]] models.transformers.transformer_stable_audio3.StableAudio3DiTModelOutput

docs/source/en/api/pipelines/diffusion_gemma.md

@@ -145,11 +145,11 @@ is not overwritten); this is the setup shown in [Usage](#usage). Drop both the `
 
 ## Adaptive stopping
 
-A block usually converges before all `num_inference_steps` are spent, so by default the pipeline leaves a block's
-denoising loop early once every example's argmax prediction is stable for `stability_threshold` steps and the mean
-per-token entropy falls below `confidence_threshold` (`0.005`, the value used by the released checkpoint). This roughly
-halves the number of decoder forwards at matched quality and is the largest single throughput lever. Pass
-`confidence_threshold=None` to always run the full `num_inference_steps`:
+A block usually converges before all `num_inference_steps` are spent, so by default the pipeline freezes each batch
+example once its argmax prediction is stable for `stability_threshold` steps and its mean per-token entropy falls below
+`confidence_threshold` (`0.005`, the value used by the released checkpoint). The denoising loop ends once every example
+is frozen. This roughly halves the number of decoder forwards at matched quality and is the largest single throughput
+lever. Pass `confidence_threshold=None` to always run the full `num_inference_steps`:
 
 ```py
 output = pipe(prompt="Why is the sky blue?", gen_length=256, confidence_threshold=None)  # disable adaptive stopping

docs/source/en/api/pipelines/kandinsky5_video.md

@@ -13,7 +13,7 @@ specific language governing permissions and limitations under the License.
 
 Kandinsky 5.0 Lite line-up of lightweight video generation models (2B parameters) that ranks #1 among open-source models in its class. It outperforms larger models and offers the best understanding of Russian concepts in the open-source ecosystem.
 
-Kandinsky 5.0 Pro line-up of large high quality video generation models (19B parameters). It offers high qualty generation in HD and more generation formats like I2V.
+Kandinsky 5.0 Pro line-up of large high quality video generation models (19B parameters). It offers high quality generation in HD and more generation formats like I2V.
 
 The model introduces several key innovations:
 - **Latent diffusion pipeline** with **Flow Matching** for improved training stability
@@ -54,7 +54,7 @@ Kandinsky 5.0 T2V Lite:
 ### Basic Text-to-Video Generation
 
 #### Pro
-**⚠️ Warning!** all Pro models should be infered with pipeline.enable_model_cpu_offload()  
+**⚠️ Warning!** all Pro models should be inferred with pipeline.enable_model_cpu_offload()  
 ```python
 import torch
 from diffusers import Kandinsky5T2VPipeline
@@ -65,7 +65,7 @@ model_id = "kandinskylab/Kandinsky-5.0-T2V-Pro-sft-5s-Diffusers"
 pipe = Kandinsky5T2VPipeline.from_pretrained(model_id, dtype=torch.bfloat16)
 
 pipe = pipe.to("cuda")
-pipeline.transformer.set_attention_backend("flex")                            # <--- Set attention bakend to Flex
+pipeline.transformer.set_attention_backend("flex")                            # <--- Set attention backend to Flex
 pipeline.enable_model_cpu_offload()                                           # <--- Enable cpu offloading for single GPU inference
 pipeline.transformer.compile(mode="max-autotune-no-cudagraphs", dynamic=True) # <--- Compile with max-autotune-no-cudagraphs
 
@@ -126,7 +126,7 @@ pipe = pipe.to("cuda")
 
 pipe.transformer.set_attention_backend(
     "flex"
-)                                       # <--- Set attention bakend to Flex
+)                                       # <--- Set attention backend to Flex
 pipe.transformer.compile(
     mode="max-autotune-no-cudagraphs", 
     dynamic=True
@@ -149,7 +149,7 @@ export_to_video(output, "output.mp4", fps=24, quality=9)
 ```
 
 ### Diffusion Distilled model
-**⚠️ Warning!** all nocfg and diffusion distilled models should be infered wothout CFG (```guidance_scale=1.0```):
+**⚠️ Warning!** all nocfg and diffusion distilled models should be inferred without CFG (```guidance_scale=1.0```):
 
 ```python
 model_id = "kandinskylab/Kandinsky-5.0-T2V-Lite-distilled16steps-5s-Diffusers"
@@ -167,7 +167,7 @@ export_to_video(output, "output.mp4", fps=24, quality=9)
 
 
 ### Basic Image-to-Video Generation
-**⚠️ Warning!** all Pro models should be infered with pipeline.enable_model_cpu_offload()  
+**⚠️ Warning!** all Pro models should be inferred with pipeline.enable_model_cpu_offload()  
 ```python
 import torch
 from diffusers import Kandinsky5T2VPipeline
@@ -178,7 +178,7 @@ model_id = "kandinskylab/Kandinsky-5.0-I2V-Pro-sft-5s-Diffusers"
 pipe = Kandinsky5T2VPipeline.from_pretrained(model_id, dtype=torch.bfloat16)
 
 pipe = pipe.to("cuda")
-pipeline.transformer.set_attention_backend("flex")                            # <--- Set attention bakend to Flex
+pipeline.transformer.set_attention_backend("flex")                            # <--- Set attention backend to Flex
 pipeline.enable_model_cpu_offload()                                           # <--- Enable cpu offloading for single GPU inference
 pipeline.transformer.compile(mode="max-autotune-no-cudagraphs", dynamic=True) # <--- Compile with max-autotune-no-cudagraphs