Skip to content

Commit 408ce8d

Browse files
tangyanftangyanfei.8dg845github-actions[bot]
authored
feat: add JoyImage edit plus (#14032)
* feat: add image edit plus * refactor: remove debug code * fix: address review issues for JoyImage Edit Plus - Remove einops dependency: replace rearrange with reshape/permute - Remove sglang-specific code from transformer forward - Remove unused import inspect from transformer - Fix hardcoded device_type="cuda" to use device.type - Simplify scheduler sigma math: delegate to retrieve_timesteps - Remove unused enable_denormalization parameter - Fix callback latents variable binding - Fix output_type="pt" to return stacked tensor - Set return_dict default to True in transformer forward - Add dummy objects for JoyImageEditPlus classes - Add transformer and pipeline test files * fix: add missing newline at end of pipeline_output.py * fix: add missing newline at end of pipeline_output.py * doc: add joyimage-edit-plus doc * refactor: update code format * refactor: update code formate * refactor: merge edit-plus conversion script into convert_joyimage_edit_to_diffusers.py JoyImage Edit and Edit Plus share identical VAE and transformer weight layouts — only the target model class differs. Consolidate both into a single script with a --model_type flag (edit | edit_plus) instead of maintaining two nearly-duplicate files. * revert: remove unnecessary attention_mask change from transformer_joyimage.py * Apply style fixes * fix: remove stale Copied-from annotations that diverged from source * fix: address CI check failures for edit-plus PR - Add missing Returns section to JoyImageEditPlusTransformer3DModel.forward docstring - Fix alphabetical ordering of dummy classes in dummy_pt_objects.py * fix: use VAE dtype instead of float32 cast in edit-plus pipeline Replace `torch.autocast(dtype=torch.float32)` + `.float()` with `.to(self.vae.dtype)` for both VAE encode and decode calls. The previous approach caused dtype mismatch (float32 input vs bfloat16 bias) on CPU where autocast does not automatically cast conv weights, breaking the CI `test_layerwise_casting_inference` test. * refactor: address dg845 review comments on edit-plus Transformer: - Simplify RoPE by removing dead txt_freqs path - Fix patch_size type hint to list[int] - Use self.config.xxx instead of explicit attributes Pipeline: - Merge VaeImageProcessor into JoyImageEditImageProcessor - Inline normalize/denormalize latents at call sites - Use image_processor.preprocess() for reference images - Remove duplicate resize_center_crop in prepare_latents - Rename padded_latents to latents in __call__ - Support pre-computed latents parameter - Remove redundant padding token zeroing (attention mask suffices) - Use image_processor.postprocess() for output conversion - Return (image,) tuple when return_dict=False * fix: guard against division-by-zero in CFG norm rescaling Add .clamp_min(1e-6) to noise_norm to prevent NaN when comb_pred is all-zeros (e.g. padded patches), matching pipeline_joyimage_edit.py. * style: run ruff format on edit-plus pipeline * refactor: address dg845 second-round review comments - Fix vae_image_processor -> image_processor in docs - Remove empty __init__ from AttnProcessor - Restore # Copied from on retrieve_timesteps and sync with source - Add check_inputs method for basic input validation - Remove unnecessary test skip for test_from_save_pretrained_dtype_inference * Apply style fixes --------- Co-authored-by: tangyanfei.8 <tangyanfei.8@jd.com> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 570b6a7 commit 408ce8d

16 files changed

Lines changed: 1906 additions & 104 deletions

‎docs/source/en/_toctree.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -353,6 +353,8 @@
353353
title: HunyuanVideoTransformer3DModel
354354
- local: api/models/ideogram4_transformer2d
355355
title: Ideogram4Transformer2DModel
356+
- local: api/models/transformer_joyimage_edit_plus
357+
title: JoyImageEditPlusTransformer3DModel
356358
- local: api/models/transformer_joyimage
357359
title: JoyImageEditTransformer3DModel
358360
- local: api/models/krea2_transformer2d
@@ -555,6 +557,8 @@
555557
title: InstructPix2Pix
556558
- local: api/pipelines/joyimage_edit
557559
title: JoyImage Edit
560+
- local: api/pipelines/joyimage_edit_plus
561+
title: JoyImage Edit Plus
558562
- local: api/pipelines/kandinsky
559563
title: Kandinsky 2.1
560564
- local: api/pipelines/kandinsky_v22

‎docs/source/en/api/models/transformer_joyimage_edit_plus.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
<!--Copyright 2025 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# JoyImageEditPlusTransformer3DModel
14+
15+
The model can be loaded with the following code snippet.
16+
17+
```python
18+
from diffusers import JoyImageEditPlusTransformer3DModel
19+
20+
transformer = JoyImageEditPlusTransformer3DModel.from_pretrained("jdopensource/JoyAI-Image-Edit-Plus-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16)
21+
```
22+
23+
## JoyImageEditPlusTransformer3DModel
24+
25+
[[autodoc]] JoyImageEditPlusTransformer3DModel
26+
27+
## Transformer2DModelOutput
28+
29+
[[autodoc]] models.modeling_outputs.Transformer2DModelOutput

‎docs/source/en/api/pipelines/joyimage_edit_plus.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<!--Copyright 2025 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# JoyAI-Image-Edit-Plus
14+
15+
[JoyAI-Image](https://github.com/jd-opensource/JoyAI-Image) is a unified multimodal foundation model for image understanding, text-to-image generation, and instruction-guided image editing. It combines an 8B Multimodal Large Language Model (MLLM) with a 16B Multimodal Diffusion Transformer (MMDiT).
16+
17+
JoyAI-Image-Edit-Plus is a multi-image instruction-guided editing model that accepts **multiple reference images** and a text instruction to generate a new image that combines elements from the references according to the instruction. It supports 1–5 reference images per sample.
18+
19+
| Model | Description | Download |
20+
|:-----:|:-----------:|:--------:|
21+
| JoyAI-Image-Edit-Plus | Multi-image instruction-guided editing with element composition from multiple references | [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Image-Edit-Plus-Diffusers) |
22+
23+
```python
24+
import torch
25+
from PIL import Image
26+
from diffusers import JoyImageEditPlusPipeline
27+
28+
pipeline = JoyImageEditPlusPipeline.from_pretrained(
29+
"jdopensource/JoyAI-Image-Edit-Plus-Diffusers", torch_dtype=torch.bfloat16
30+
)
31+
pipeline.to("cuda")
32+
33+
images = [
34+
Image.open("reference_0.png").convert("RGB"),
35+
Image.open("reference_1.png").convert("RGB"),
36+
]
37+
38+
target_h, target_w = pipeline.image_processor.get_default_height_width(images[-1])
39+
40+
output = pipeline(
41+
images=images,
42+
prompt="Combine the person from the second image with the scene from the first image.",
43+
negative_prompt="low quality, blurry, deformed",
44+
height=target_h,
45+
width=target_w,
46+
num_inference_steps=30,
47+
guidance_scale=4.0,
48+
generator=torch.Generator("cuda").manual_seed(42),
49+
).images[0]
50+
output.save("joyimage_edit_plus_output.png")
51+
```
52+
53+
## JoyImageEditPlusPipeline
54+
55+
[[autodoc]] JoyImageEditPlusPipeline
56+
- all
57+
- __call__
58+
59+
## JoyImageEditPlusPipelineOutput
60+
61+
[[autodoc]] pipelines.joyimage.pipeline_output.JoyImageEditPlusPipelineOutput

0 commit comments

Comments
 (0)