Hi there,
I'm testing INT8 models using ROCm and Triton backend. In this scenario, using Krea 2 leads me to a memory fault crash. This only happens when enabling Triton backend.
Tried using Flash Attention, Pytorch Cross Attention and Sub Quad Optimization, same result.
Disabling the Triton backend aparently works fine, just terribly slow, painfully slower than FP8.
Model used: https://huggingface.co/Comfy-Org/Krea-2/blob/main/diffusion_models/krea2_turbo_int8_convrot.safetensors
[ 5603.532594] amdgpu 0000:c4:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:94)
[ 5603.532595] amdgpu 0000:c4:00.0: Process python3 pid 135257 thread python3 pid 135257
[ 5603.532596] amdgpu 0000:c4:00.0: in page starting at address 0x00007f4d8a007000 from client 10
[ 5603.532598] amdgpu 0000:c4:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00801031
[ 5603.532598] amdgpu 0000:c4:00.0: Faulty UTCL2 client ID: TCP (0x8)
[ 5603.532599] amdgpu 0000:c4:00.0: MORE_FAULTS: 0x1
[ 5603.532600] amdgpu 0000:c4:00.0: WALKER_ERROR: 0x0
[ 5603.532601] amdgpu 0000:c4:00.0: PERMISSION_FAULTS: 0x3
[ 5603.532602] amdgpu 0000:c4:00.0: MAPPING_ERROR: 0x0
[ 5603.532603] amdgpu 0000:c4:00.0: RW: 0x0
ComfyUI logs:
comfyui.log
Any extra analysis, let me know, happy to provide information.
ROCM version: 7.2.0
Triton version: 3.7.1
Comfy Kitchen version: 0.2.16
ComfyUI version: 0.27.0
Footnote:
I'm also using INT8 CLIP for text generation, and this seems to work fine. Probably 20% faster than FP8. It doesn't crash.
INT8 Clip used: https://huggingface.co/Winnougan/Comfy-Qwen3-VL-INT8/blob/main/qwen3vl_4b_int8_convrot.safetensors
FP8 Clip used: https://huggingface.co/Comfy-Org/Qwen3-VL/blob/8bb77f7ad334bc594cd68c0c48e868abd1e0f719/text_encoders/qwen3vl_4b_fp8_scaled.safetensors
Hi there,
I'm testing INT8 models using ROCm and Triton backend. In this scenario, using Krea 2 leads me to a memory fault crash. This only happens when enabling Triton backend.
Tried using Flash Attention, Pytorch Cross Attention and Sub Quad Optimization, same result.
Disabling the Triton backend aparently works fine, just terribly slow, painfully slower than FP8.
Model used: https://huggingface.co/Comfy-Org/Krea-2/blob/main/diffusion_models/krea2_turbo_int8_convrot.safetensors
[ 5603.532594] amdgpu 0000:c4:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:94)
[ 5603.532595] amdgpu 0000:c4:00.0: Process python3 pid 135257 thread python3 pid 135257
[ 5603.532596] amdgpu 0000:c4:00.0: in page starting at address 0x00007f4d8a007000 from client 10
[ 5603.532598] amdgpu 0000:c4:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00801031
[ 5603.532598] amdgpu 0000:c4:00.0: Faulty UTCL2 client ID: TCP (0x8)
[ 5603.532599] amdgpu 0000:c4:00.0: MORE_FAULTS: 0x1
[ 5603.532600] amdgpu 0000:c4:00.0: WALKER_ERROR: 0x0
[ 5603.532601] amdgpu 0000:c4:00.0: PERMISSION_FAULTS: 0x3
[ 5603.532602] amdgpu 0000:c4:00.0: MAPPING_ERROR: 0x0
[ 5603.532603] amdgpu 0000:c4:00.0: RW: 0x0
ComfyUI logs:
comfyui.log
Any extra analysis, let me know, happy to provide information.
ROCM version: 7.2.0
Triton version: 3.7.1
Comfy Kitchen version: 0.2.16
ComfyUI version: 0.27.0
Footnote:
I'm also using INT8 CLIP for text generation, and this seems to work fine. Probably 20% faster than FP8. It doesn't crash.
INT8 Clip used: https://huggingface.co/Winnougan/Comfy-Qwen3-VL-INT8/blob/main/qwen3vl_4b_int8_convrot.safetensors
FP8 Clip used: https://huggingface.co/Comfy-Org/Qwen3-VL/blob/8bb77f7ad334bc594cd68c0c48e868abd1e0f719/text_encoders/qwen3vl_4b_fp8_scaled.safetensors