Skip to content

Commit 0ceddf7

Browse files
authored
[docs] add docs for JoyAI-Image-Edit (#13726)
add docs
1 parent a59f359 commit 0ceddf7

3 files changed

Lines changed: 118 additions & 0 deletions

File tree

docs/source/en/_toctree.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,8 @@
372372
title: HunyuanVideo15Transformer3DModel
373373
- local: api/models/hunyuan_video_transformer_3d
374374
title: HunyuanVideoTransformer3DModel
375+
- local: api/models/transformer_joyimage
376+
title: JoyImageEditTransformer3DModel
375377
- local: api/models/latte_transformer3d
376378
title: LatteTransformer3DModel
377379
- local: api/models/longcat_image_transformer2d
@@ -560,6 +562,8 @@
560562
title: HunyuanImage2.1
561563
- local: api/pipelines/pix2pix
562564
title: InstructPix2Pix
565+
- local: api/pipelines/joyimage_edit
566+
title: JoyImage Edit
563567
- local: api/pipelines/kandinsky
564568
title: Kandinsky 2.1
565569
- local: api/pipelines/kandinsky_v22
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
<!--Copyright 2025 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# JoyImageEditTransformer3DModel
14+
15+
The model can be loaded with the following code snippet.
16+
17+
```python
18+
from diffusers import JoyImageEditTransformer3DModel
19+
20+
transformer = JoyImageEditTransformer3DModel.from_pretrained("jdopensource/JoyAI-Image-Edit-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16)
21+
```
22+
23+
## JoyImageEditTransformer3DModel
24+
25+
[[autodoc]] JoyImageEditTransformer3DModel
26+
27+
## Transformer2DModelOutput
28+
29+
[[autodoc]] models.modeling_outputs.Transformer2DModelOutput
Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
<!--Copyright 2025 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# JoyAI-Image-Edit
14+
15+
[JoyAI-Image](https://github.com/jd-opensource/JoyAI-Image) is a unified multimodal foundation model for image understanding, text-to-image generation, and instruction-guided image editing. It combines an 8B Multimodal Large Language Model (MLLM) with a 16B Multimodal Diffusion Transformer (MMDiT). A central principle of JoyAI-Image is the closed-loop collaboration between understanding, generation, and editing.
16+
17+
JoyAI-Image-Edit supports general image editing as well as spatial editing capabilities including object move, object rotation, and camera control.
18+
19+
| Model | Description | Download |
20+
|:-----:|:-----------:|:--------:|
21+
| JoyAI-Image-Edit | Instruction-guided image editing with precise and controllable spatial manipulation | [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Image-Edit-Diffusers) |
22+
23+
```python
24+
import torch
25+
from diffusers import JoyImageEditPipeline
26+
from diffusers.utils import load_image
27+
28+
pipeline = JoyImageEditPipeline.from_pretrained(
29+
"jdopensource/JoyAI-Image-Edit-Diffusers", torch_dtype=torch.bfloat16
30+
)
31+
pipeline.to("cuda")
32+
33+
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
34+
prompt = "Add wings to the astronaut."
35+
36+
output = pipeline(
37+
image=image,
38+
prompt=prompt,
39+
num_inference_steps=40,
40+
guidance_scale=4.0,
41+
generator=torch.Generator("cuda").manual_seed(0),
42+
).images[0]
43+
output.save("joyimage_edit_output.png")
44+
```
45+
46+
## Spatial editing
47+
48+
JoyAI-Image supports three spatial editing prompt patterns: **Object Move**, **Object Rotation**, and **Camera Control**. For best results, follow the prompt templates below as closely as possible. For more information, refer to [SpatialEdit](https://github.com/EasonXiao-888/SpatialEdit).
49+
50+
### Object Move
51+
52+
Move a target object into a specified region marked by a red box in the input image.
53+
54+
```text
55+
Move the <object> into the red box and finally remove the red box.
56+
```
57+
58+
### Object Rotation
59+
60+
Rotate an object to a specific canonical view. Supported `<view>` values: `front`, `right`, `left`, `rear`, `front right`, `front left`, `rear right`, `rear left`.
61+
62+
```text
63+
Rotate the <object> to show the <view> side view.
64+
```
65+
66+
### Camera Control
67+
68+
Change the camera viewpoint while keeping the 3D scene unchanged.
69+
70+
```text
71+
Move the camera.
72+
- Camera rotation: Yaw {y_rotation}°, Pitch {p_rotation}°.
73+
- Camera zoom: in/out/unchanged.
74+
- Keep the 3D scene static; only change the viewpoint.
75+
```
76+
77+
## JoyImageEditPipeline
78+
79+
[[autodoc]] JoyImageEditPipeline
80+
- all
81+
- __call__
82+
83+
## JoyImageEditPipelineOutput
84+
85+
[[autodoc]] pipelines.joyimage.pipeline_output.JoyImageEditPipelineOutput

0 commit comments

Comments
 (0)