[BUG][TorchFX] uint8 zero-point tensors cause wraparound in per-channel dequantization of asymmetric weights #4311

🐛 Describe the bug

Asymmetric per-channel weight quantization on the TorchFX backend gives wrong outputs even with backend_params={"compress_weights": False}.

compress_weights is a TorchFX backend parameter of nncf.quantize() (default True). Setting it to False skips compress_post_quantize_transformation and runs fq_weights_transformation instead. It does not disable quantization and is unrelated to nncf.compress_weights().

Reproduced on commit 2dec7db5b405cde920aab48e19290689e5657d4d.

Console output of the example below (NNCF warnings and progress bars removed):

float: [[0.6849863529205322, 1.5280396938323975]]
asymmetric, per_channel=True: [[2.0007810592651367, 1.5267161130905151]]
asymmetric, per_channel=False: [[0.6850670576095581, 1.5294520854949951]]

Per-tensor quantization is fine. Per-channel row 0 is wrong (2.0008 instead of about 0.6856).

What I traced:

1. get_scale_zp_from_input_low_input_high returns a uint8 zero-point tensor for an unsigned range (int8 for a signed range).
2. FXMinMaxAlgoBackend._create_quantizer (L238-L255) assigns it to fakequantizer.zero_point unchanged. insert_quantize_dequantize_node passes it unchanged to the per-channel Q/DQ ops (transformations.py L397). The per-tensor branch converts it with int(quantizer.zero_point) (L408), which explains why per-tensor is unaffected.
3. In torch 2.14.0, torch/ao/quantization/fx/_decomposed.py::dequantize_per_channel computes (input - zero_points) * scales and only then converts to the output dtype. With both input and zero_points in uint8, negative differences wrap modulo 256.

In the example, row 0 has codes [153, 255, 0, 17] and zero point 42:

correct differences:          [111, 213, -42, -25]
differences computed in uint8: [111, 213, 214, 231]
correct dequantized row:      [0.29611561, 0.56822181, -0.11204374, -0.06669270]
actual dequantized row:       [0.29611561, 0.56822181,  0.57088953,  0.61624056]

The model already outputs [[2.05146742, 1.52671611]] before fq_weights_transformation runs, so the wraparound comes from the Q/DQ pair inserted during quantization, not from the final weight transformation.

Operator-level check with an NNCF-produced zero point:

import torch
import torch.ao.quantization.fx._decomposed  # registers the quantized_decomposed ops
from nncf.torch.quantization.quantize_functions import get_scale_zp_from_input_low_input_high

scale, zp = get_scale_zp_from_input_low_input_high(0, 255, torch.tensor([-0.5]), torch.tensor([0.5]))
scale, zp = scale.reshape(1), zp.reshape(1)  # zp: tensor([128], dtype=torch.uint8)
codes = torch.tensor([[0, 128, 255]], dtype=torch.uint8)
dequantize = torch.ops.quantized_decomposed.dequantize_per_channel.default

print(dequantize(codes, scale, zp, 0, 0, 255, torch.uint8))                   # [[ 0.5020, 0.0000, 0.4980]]
print(dequantize(codes, scale, zp.to(torch.int32), 0, 0, 255, torch.uint8))   # [[-0.5020, 0.0000, 0.4980]]
print((codes.float() - zp.float().reshape(-1, 1)) * scale.reshape(-1, 1))     # [[-0.5020, 0.0000, 0.4980]]

To confirm the cause, I temporarily patched _create_quantizer to cast the zero point to int32, I wrapped FXMinMaxAlgoBackend._create_quantizer to set quantizer.zero_point = quantizer.zero_point.to(torch.int32) after the original call. With compress_weights=False, the per-channel output becomes [[0.68560100, 1.52671611]], matching the float reconstruction from the same codes. The symmetric output does not change ([[0.68455070, 1.52998495]]).

I have not yet determined the correct zero-point dtype at the TorchFX Q/DQ boundary or whether the shared helper should change, since get_scale_zp_from_input_low_input_high has other users. I also have not checked signed nonzero zero points.

I searched open and closed issues and PRs (zero point dtype, uint8, wraparound, dequantize_per_channel, asymmetric FX) and did not find an existing report.

I would like to investigate the appropriate zero-point dtype at the TorchFX Q/DQ boundary and add regression coverage.

Environment

OS                  Fedora Linux 44, x86_64
Python              3.13.16
Install             git, editable (uv pip install -e .) at commit 2dec7db5b405cde920aab48e19290689e5657d4d
RAM                 ~15.3 GiB
CPU                 13th Gen Intel Core i5-13450HX
CUDA                None used (PyTorch CUDA 13.0 build; torch.cuda.is_available() == False)

nncf.__version__    3.5.0
torch               2.14.0+cu130
torchao             0.18.0
torchvision         0.29.0
numpy               2.5.4
openvino            2026.4.1 (installed, not used to run this example)
onnx                1.23.0   (installed, not used to run this example)

Minimal Reproducible Example

import torch
import nncf

from nncf.common.quantization.structs import QuantizationScheme
from nncf.quantization.advanced_parameters import (
    AdvancedQuantizationParameters,
    QuantizationParameters,
)

torch.manual_seed(0)
model = torch.nn.Linear(4, 2, bias=False).eval()
with torch.no_grad():
    model.weight.add_(0.3)

x = torch.ones(1, 4)
exported = torch.export.export(model, (x,)).module()
dataset = nncf.Dataset([x.clone() for _ in range(4)])

print("float:", model(x).tolist())

for per_channel in (True, False):
    params = AdvancedQuantizationParameters(
        weights_quantization_params=QuantizationParameters(
            mode=QuantizationScheme.ASYMMETRIC, per_channel=per_channel
        ),
        disable_bias_correction=True,
        backend_params={"compress_weights": False},
    )
    quantized = nncf.quantize(
        exported,
        dataset,
        subset_size=4,
        advanced_parameters=params,
    )
    print(f"asymmetric, per_channel={per_channel}:", quantized(x).tolist())

Are you going to submit a PR?

• Yes I'd like to help by submitting a PR!