Hello everyone, I am the author of ConvRot. First of all, I want to thank the friends in the Comfy community for their love and support for ConvRot. Initially, ConvRot was designed for INT4 quantization because QuaRot W4A4 encountered accuracy issues on DiT models. We discovered that the DiT model has a unique type of row-wise outlier, which causes the global rotation matrix in QuaRot to accumulate these row-wise outliers. This accumulation not only fails to distribute and suppress the outliers evenly but instead amplifies them, ultimately leading to accuracy loss.
The first innovation of ConvRot is the use of grouped rotation instead of global rotation, which brings two benefits. The first benefit is that the computational complexity is reduced from O(n²) to O(n), enabling extreme acceleration. The second benefit is that it can reduce the size of the rotation matrix multiplication and accumulation operations, thereby improving quantization accuracy and addressing the issue of rotation algorithm accuracy failure in the image and video domain.
The second innovation is the definition of "column imbalance" to measure the ability of the rotation matrix to distribute outliers. We designed a regular Hadamard rotation matrix with optimal column imbalance, fundamentally solving the accuracy failure problem caused by outlier amplification in image and video generation. The regular Hadamard rotation matrix must be a power of 4, so the group sizes you can choose include 16, 64, 256, 1024, etc. Regarding group size, it seems that ComfyUI is still restricting the relationship between group size and input shape, but this is unnecessary. You don't need to worry about whether it fits the input shape of the linear layer because we have already handled dimension mismatch issues with padding inside the algorithm. Based on our experience, in most scenarios, a group size of 256 is sufficient to achieve a good balance between performance and accuracy. If you are pursuing higher accuracy, you can try 1024.
Finally, the Comfy community currently only integrates ConvRot W8A8. I want to emphasize that the real advantage of ConvRot lies in W4A4. I look forward to the Comfy community using ConvRot W4A4 to provide users with an even more outstanding performance experience. Moreover, ConvRot is not only applicable to DiT models but also to LLMs, VLMs, and even Unets, and it is not limited to integer quantization, it is also suitable for floating-point quantization.
Hello everyone, I am the author of ConvRot. First of all, I want to thank the friends in the Comfy community for their love and support for ConvRot. Initially, ConvRot was designed for INT4 quantization because QuaRot W4A4 encountered accuracy issues on DiT models. We discovered that the DiT model has a unique type of row-wise outlier, which causes the global rotation matrix in QuaRot to accumulate these row-wise outliers. This accumulation not only fails to distribute and suppress the outliers evenly but instead amplifies them, ultimately leading to accuracy loss.
The first innovation of ConvRot is the use of grouped rotation instead of global rotation, which brings two benefits. The first benefit is that the computational complexity is reduced from O(n²) to O(n), enabling extreme acceleration. The second benefit is that it can reduce the size of the rotation matrix multiplication and accumulation operations, thereby improving quantization accuracy and addressing the issue of rotation algorithm accuracy failure in the image and video domain.
The second innovation is the definition of "column imbalance" to measure the ability of the rotation matrix to distribute outliers. We designed a regular Hadamard rotation matrix with optimal column imbalance, fundamentally solving the accuracy failure problem caused by outlier amplification in image and video generation. The regular Hadamard rotation matrix must be a power of 4, so the group sizes you can choose include 16, 64, 256, 1024, etc. Regarding group size, it seems that ComfyUI is still restricting the relationship between group size and input shape, but this is unnecessary. You don't need to worry about whether it fits the input shape of the linear layer because we have already handled dimension mismatch issues with padding inside the algorithm. Based on our experience, in most scenarios, a group size of 256 is sufficient to achieve a good balance between performance and accuracy. If you are pursuing higher accuracy, you can try 1024.
Finally, the Comfy community currently only integrates ConvRot W8A8. I want to emphasize that the real advantage of ConvRot lies in W4A4. I look forward to the Comfy community using ConvRot W4A4 to provide users with an even more outstanding performance experience. Moreover, ConvRot is not only applicable to DiT models but also to LLMs, VLMs, and even Unets, and it is not limited to integer quantization, it is also suitable for floating-point quantization.