Recurrent Residual Quantization (RRQ) for LLMs

[luoyu-intel]

PR: Recurrent Residual Quantization (RRQ) for LLMs

Branch: feat/rrq-phase1 → main
Scope: 5 commits, 27 files, +4,826 / −6

Recurrent Residual Quantization (RRQ) quantizes each weight tensor into K sequential
INT2 planes (1 base + K−1 residual planes). The base plane is a standard AutoRound
INT2 export; the residual planes are packed together into a new auto_round:rrq
artifact. At inference the effective precision is selected dynamically
(2 / 4 / 6 / 8-bit) — and, as of the latest commit, per-layer mixed precision —
from a single checkpoint, without re-quantizing.

---

1. Motivation

• One checkpoint, many precisions. A single RRQ export serves 2/4/6/8-bit by
  loading more or fewer residual planes, instead of shipping a separate model per
  bit-width.
• Standard-compatible base. The base plane is an ordinary W2A16 model that existing
  runtimes can load as-is; RRQ adds residual planes on top.
• Quality parity with AutoRound. Each plane is quantized with the same opt-RTN /
  SignRound machinery as ordinary AutoRound, so RRQ is a fair, drop-in extension rather
  than a weaker parallel path.

---

2. Design

weight W  ─►  plane 0 (base, INT2)         ── standard auto_round export
             + plane 1 (residual, INT2) ┐
             + plane 2 (residual, INT2) ├─ packed together as auto_round:rrq
             + plane 3 (residual, INT2) ┘

effective bits = active_planes × 2   (base=2, +1=4, +2=6, +3=8)

• Quantization. Each plane quantizes the residual left by the accumulated prefix of
  all previous planes. The base plane uses imatrix-weighted opt-RTN (bit-exact with
  a standard W2A16 base); residual planes seed their scale search with
  search_optimized_init_scale. With iters>0, per-plane sign-SGD (SignRound)
  tuning optimizes value/min_scale/max_scale through the STE path while freezing the
  completed prefix.
• Storage. Packed INT2, reusing the W2A16 QuantLinear pack/forward path.
  • Base: qweight / scales / qzeros (standard auto_round).
  • Residual: qweight_{1..3} / scales_{1..3} / qzeros_{1..3} in one auto_round:rrq
    artifact, tagged quant_method = "auto-round-rrq".
• Inference. RRQLinear computes the base output, then accumulates the first
  active_planes − 1 residual outputs. set_rrq_bits(model, bits) switches precision
  uniformly; set_rrq_random_residual(...) / load_rrq_model(residual_fraction=...)
  assign precision per layer.

---

3. Commit breakdown

Commit	Title	Highlights
3299eb3e	Phase 1 — RRQ (packed INT2 2+2+2+2)	RRQConfig, RRQRTNQuantizer, RRQFormat, RRQLinear, load_rrq_model; export helpers; format/back-end wiring; GGUF/MLX fail-fast guards; 23 tests.
6afb9a7f	Phase 2 — generate_rrq_residual	Build the 3 residual planes from an existing INT2 base + FP weights without re-quantizing the base; local dir / HF name support; config fail-fast; 5 tests.
5c7c12e5	Phase 3 — per-plane sign-SGD tuning	RRQSignRoundQuantizer with sequential AutoRound rounds; freeze completed prefix; route OPT configs through the calibrated compressor; keep RTN for iters=0.
0fd545f3	fix — accumulate optimized residual prefix	Each round freezes the cumulative sum of all previous planes (not just the previous one); reconstruction regression test.
b38849a9	feat — match AutoRound quality + configurable mixed precision	imatrix-weighted opt-RTN base (bit-exact); init_scale seeding; need_calib always; SignRound defaults (iters=200); set_rrq_random_residual + residual_fraction; new test.

---

4. Public API

# --- quantize ---
from auto_round import AutoRound
ar = AutoRound(model, scheme=..., )          # RRQConfig(num_residual_planes=1|3, iters=...)
ar.quantize()
ar.save_quantized("./rrq-base",     format="auto_round")       # base plane (standard INT2)
ar.save_quantized("./rrq-residual", format="auto_round:rrq")   # residual planes

# --- or build residual from an existing base (Phase 2) ---
from auto_round import generate_rrq_residual
generate_rrq_residual("./rrq-base", raw_model="Qwen/Qwen3-0.6B", output_dir="./rrq-residual")

# --- load & run ---
from auto_round.inference.rrq_model import load_rrq_model
from auto_round.inference.rrq_linear import set_rrq_bits, set_rrq_random_residual

m = load_rrq_model("./rrq-base", "./rrq-residual", active_bits=4, device="xpu")  # uniform
set_rrq_bits(m, 6)                                                               # switch precision

# mixed precision: 50% of layers at 4-bit, rest at 2-bit (~3-bit effective), seeded
m = load_rrq_model("./rrq-base", "./rrq-residual", device="xpu",
                   residual_fraction=0.5, residual_seed=0,
                   residual_high_bits=4, residual_low_bits=2)
set_rrq_random_residual(m, fraction=0.5, seed=3, high_bits=4, low_bits=2)

---

5. Changed files (cumulative vs main)

Area	Files
Algorithm	algorithms/quantization/rrq/{__init__,config,quantizer}.py, algorithms/registry.py
Export	export/export_to_autoround/export_to_rrq.py, export/formats/backends/rrq.py, export/formats/backends/__init__.py
Inference	inference/rrq_linear.py, inference/rrq_model.py, inference/backend.py
Integration	auto_round/__init__.py, autoround.py, compressors/base.py, compressors/model_free.py, utils/common.py
Fail-fast guards	export/export_to_gguf/conversion/base.py, export/export_to_mlx/export.py
Bug fix	auto_round_extension/torch/qlinear_torch.py (asym pack self.device → device)
Tests	test/unit/test_cpu/algorithms/test_rrq.py (36 tests)
Tooling	test_rrq_qwen3_06b.py, test_rrq_lm_eval.py
Docs	docs/rrq_rfc.md, docs/rrq_rfc_CN.md, docs/rrq_progress_CN.md, docs/PR_rrq_phase1.md, docs/rrq_pr_summary.md, .gitignore

---

6. Validation

6a. Base-plane parity

The RRQ base plane is bit-exact with a standard W2A16 imatrix-weighted opt-RTN base
at matched calibration (batch_size=8): 0 / 702 element differences.

6b. Accuracy — Phase 1, HellaSwag (Qwen3-0.6B, group_size=128, asym, XPU, limit=200)

Bits	Planes	Accuracy	Δ vs fp32
fp32	—	43.5%	—
6-bit	3	43.5%	0
8-bit	4	42.5%	−1.0pp
4-bit	2	35.5%	−8.0pp
2-bit	1	26.5%	−17.0pp

6c. Accuracy — opt-RTN, 5-task mean (Qwen3-0.6B, sym)

5-task mean = piqa / winogrande / hellaswag / arc_easy / arc_challenge. FP = 50.65.

group_size = 128 (default config). The RRQ default base is imatrix-weighted opt-RTN;
residual planes seed the scale search with init_scale. RTN shown for reference.

Bits	RTN	opt-RTN + init_scale (default)
2	35.79	41.65
4	45.42	47.54
6	49.80	48.55
8	50.99	50.28

The init_scale/opt-RTN path wins at low bits (2/4); plain RTN edges ahead at 6/8-bit
because opt-RTN's larger scales sit outside SignRound's [0,1] scale space at high
bit-widths.

group_size = 16. Finer groups recover most of the gap; 6-/8-bit reach or exceed FP.

Bits	5-task Mean (RTN)	arc_easy
2	37.33	31.90
4	49.23	57.997
6	50.72	61.24
8	50.73	60.65

6d. Mixed precision (~3-bit, arc_easy, group_size=16)

Config	arc_easy	Effective bits
base (2-bit)	31.90	2.0
random 50% residual (8-seed mean)	42.88 ± 1.58	~3.0
fixed W3A16 (opt-RTN)	43.18	3.0
full residual (4-bit)	57.997	4.0

Weight-error–based layer ranking (energy / ΔE) concentrates the budget on mlp.down_proj
and under-performs random allocation on downstream accuracy; random lands within noise of
a true 3-bit model while keeping a single checkpoint (±1.5 arc_easy variance across seeds).

6e. Storage (Phase 1, Qwen3-0.6B, fp32 = 1.40 GB)

These are full on-disk artifact sizes, not quantized-weight-only sizes. The base
model bundles the fp16 non-quantized parameters (token embeddings, layernorms,
lm_head) alongside the packed INT2 base plane; the residual model contains only the
3 packed INT2 planes (no non-quant params). So the two rows are not an apples-to-apples
"quantized weight" comparison — the base is inflated by the shared fp16 tensors.

Artifact	Size	vs fp32	Contents
Base model	420 MB	29.3%	packed INT2 base plane + fp16 embeddings/layernorms/lm_head
Residual model	337 MB	23.5%	3 × packed INT2 planes only
Combined	757 MB	53.8%	base + residual

Comparing only the quantized-weight portion, the 3 residual planes are ~3× the single
base plane (as designed); the base artifact's extra bulk is the fp16 non-quantized tensors
that any W2A16 checkpoint also carries.

6f. Unit tests

pytest test/unit/test_cpu/algorithms/test_rrq.py — 36 passed. Coverage: config
validation, packed INT2 storage, residual convergence, symmetric/asymmetric, RRQLinear
forward & precision switching, sign-SGD prefix accumulation, export buffer rename +
config attach, load validation, incremental residual generation, and the new
random-residual mixed-precision config.

---

7. Backward compatibility

• Default load_rrq_model behaviour (uniform active_bits) is unchanged when
  residual_fraction is not supplied.
• RRQConfig.iters default changed from 0 (RTN) to 200 (SignRound). Pass iters=0
  to restore pure RTN.
• num_residual_planes now accepts 1 (4-bit scheme) or 3 (2/4/6/8-bit scheme).
• GGUF / MLX export explicitly reject RRQ residual models (fail-fast, no silent
  dropping).

---

8. Known limitations & follow-ups

• Reference inference path dequantizes per plane and runs stock W2A16 matmuls — a
  correctness reference, not a fused kernel (packed-INT2 fused kernels are Phase 2+).
• Weight-only (act_bits=16); no activation quantization.
• Fixed 2-bit per plane (no 3/4-bit per-plane planes).
• Mixed-precision layer selection is random/uniform; an end-to-end sensitivity-based
  selector (imatrix-weighted output error) is a promising follow-up to beat random.

[luoyu-intel]

Qwen3-8B RRQ Accuracy Results

Experiment Setup

• Model: Qwen3-8B
• Repository branch: feat/rrq-phase1
• RRQ checkpoint base: INT2, group_size=128
• RRQ residual planes: three INT2 planes, giving uniform effective 2/4/6/8-bit modes
• Calibration dataset: NeelNanda/pile-10k
• Evaluation tasks: piqa, winogrande, hellaswag, arc_easy, arc_challenge
• Evaluation uses the complete datasets, batch size 8, --skip-fp
• Metric: acc_norm,none (primary), fallback to acc,none for tasks without acc_norm (e.g. WinoGrande)

Calibration configuration (aligned with paper auto-round-best settings):

Config	Samples	Seq Len	Batch	sym	iters	lr
bestsym	512	2048	8	True	0 / 50	2e-3

File Size Analysis

Model	Size (GB)	Compression Ratio vs FP
FP16 (original)	15.23	1.0x
W4A16 (auto-round GPTQ)	5.69	2.67x
RRQ base (INT2, 2-bit)	4.06	3.75x
RRQ base + 1 residual plane (4-bit)	5.79	2.63x
RRQ base + 2 residual planes (6-bit)	7.52	2.03x
RRQ base + 3 residual planes (8-bit)	9.25	1.65x

Notes:

• Base contains INT2 packed weights (quantizable layers) + FP16 non-quantizable layers (embeddings, layernorms, etc.)
• Residual file stores 3 separate INT2 planes together; per-plane size is ~1.73 GB
• Effective storage for 4-bit / 6-bit requires base + corresponding number of residual planes
• RRQ 4-bit (5.79 GB) is slightly larger than W4A16 (5.69 GB) due to the separate INT2 base + overhead
• RRQ 8-bit (9.25 GB) is smaller than W8A16 (~8.19 GB FP16 equivalent) only when non-quantizable layers are excluded; total is still larger

Mean Accuracy: bestsym (symmetric, aligned with paper)

All values are percentages. Differences are percentage points (pp).

Precision	RTN (iters=0)	OPT (iters=50, lr=2e-3)	Paper RTN	Paper OPT
2-bit	46.52	55.93	34.66	41.06
4-bit	69.02	69.85	70.10	70.39
6-bit	71.86	71.74	71.56	71.63
8-bit	71.73	71.74	71.63	71.65
8−6 gap	−0.13	0.00	−0.07	−0.02
Approx. 3-bit	61.11	64.76	—	—
Approx. 5-bit	70.06	70.92	—	—

The 6≈8 phenomenon is reproduced: 6-bit and 8-bit achieve nearly identical accuracy (gap ≤0.13 pp).

W4A16 Baseline (bestsym config)

Standard AutoRound W4A16 achieved 67.85% mean accuracy (pre-fix asymmetric eval, raw acc,none).

Comparison	Delta vs W4A16 (67.85)
RTN approx. 5-bit (bestsym)	+2.21 pp
OPT approx. 5-bit (bestsym)	+3.07 pp
RTN 6-bit (bestsym)	+4.01 pp
OPT 6-bit (bestsym)	+3.89 pp
RTN 8-bit (bestsym)	+3.88 pp
OPT 8-bit (bestsym)	+3.89 pp

Note: The W4A16 baseline was evaluated with the pre-fix (raw acc,none) metric; its acc_norm value may differ slightly. A direct W4A16 re-eval with acc_norm is needed for a strictly comparable comparison. The key observation is RRQ at all precision levels significantly exceeds W4A16.

Task-Level Accuracy: bestsym

All values are percentages.

Configuration	piqa	winogrande	hellaswag	arc_easy	arc_challenge	Mean
RTN 2-bit	62.35	52.33	40.35	50.80	26.79	46.52
OPT 2-bit	69.59	61.40	51.99	61.78	34.90	55.93
RTN 4-bit	76.17	66.61	72.61	76.98	52.73	69.02
OPT 4-bit	76.71	68.90	72.60	78.20	52.82	69.85
RTN 6-bit	77.31	69.30	75.01	80.77	56.91	71.86
OPT 6-bit	77.09	68.98	75.12	80.60	56.91	71.74
RTN 8-bit	77.69	68.43	74.70	81.02	56.83	71.73
OPT 8-bit	77.31	68.51	74.97	81.02	56.91	71.74
RTN approx. 3-bit	71.93	61.64	61.35	69.95	40.70	61.11
OPT approx. 3-bit	73.67	64.96	64.84	74.62	45.73	64.76
RTN approx. 5-bit	76.39	66.85	74.19	78.37	54.52	70.06
OPT approx. 5-bit	76.82	68.82	74.17	79.42	55.38	70.92

Paper Reference (Qwen3-8B, Section 5)

Configuration	piqa	winogrande	hellaswag	arc_easy	arc_challenge	Mean
RRQ RTN 6-bit	78.68	68.28	75.88	81.24	57.62	71.56
RRQ RTN 8-bit	78.73	68.54	75.88	81.31	57.50	71.63
RRQ OPT 6-bit	78.65	68.54	75.86	81.31	57.32	71.63
RRQ OPT 8-bit	78.78	68.11	76.02	81.40	57.41	71.65

Mixed-Precision Definitions

• Approx. 3-bit: 50% of RRQ layers use 4 effective bits and 50% use 2 effective bits. The layer-average precision is approximately 3 bits.
• Approx. 5-bit: 50% of RRQ layers use 6 effective bits and 50% use 4 effective bits. This corresponds to one full residual plane plus a random half of the next residual plane, with an average precision of approximately 5 bits.
• The random layer assignment is reproducible with residual_seed=42.

Key Findings

1. Symmetric quantization is critical: The 6≈8 phenomenon (6-bit accuracy ≈ 8-bit) only appears with symmetric quantization (sym=True). With asymmetric quantization (sym=False), a 0.4–0.5 pp gap persists between 6-bit and 8-bit (see earlier asymmetric runs).

2. 6≈8 saturation: With symmetric INT2 planes, the 4th plane (bits 7-8) contributes negligible accuracy gain. The 3rd plane (bits 5-6) captures essentially all useful residual information. This is consistent with the paper's observation.

3. RTN vs OPT at 6/8-bit: With symmetric quantization and 512-sample calibration, the difference between RTN and OPT at 6/8-bit is negligible (≤0.13 pp). Sign-SGD tuning provides significant benefit at 2-bit (+9.4 pp) and moderate benefit at 4-bit (+0.8 pp), but diminishing returns at 6/8-bit.

4. 2-bit advantage: Our RRQ 2-bit (46.52% RTN, 55.93% OPT) substantially exceeds the paper's reported values (34.66%/41.06%), likely due to better calibration (512 samples vs fewer) and the imatrix-enhanced opt-RTN initialization.

5. Mixed-precision (bestsym, acc_norm): Approx. 5-bit (50% at 6-bit + 50% at 4-bit) nearly matches 6-bit for both RTN (70.06 vs 71.86, −1.80 pp) and OPT (70.92 vs 71.74, −0.82 pp). Approx. 3-bit (50% at 4-bit + 50% at 2-bit) sits between 2-bit and 4-bit, with a larger OPT advantage (64.76 vs 61.11, +3.65 pp) than at 6/8-bit.

The random mixed-precision CLI options are implemented in test_rrq_lm_eval.py.

[luoyu-intel]

implementation of #2300

[AutoRoundBot]

/azp run Unit-Test-CUDA-AutoRound

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

🟡 Changes recommended

load_rrq_model can silently leave some packed base layers as uninitialized/random nn.Linear weights when residual planes are missing or skipped, which is correctness-critical for inference.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Recurrent Residual Quantization (RRQ) to AutoRound, enabling a single INT2-base checkpoint plus packed INT2 residual planes to support dynamic 2/4/6/8-bit (and mixed-precision) inference via a new auto_round:rrq residual artifact and corresponding loader/runtime modules.

Changes:

• Introduces RRQ algorithm config + quantizers (RTN and per-plane SignRound tuning) and registers it in the algorithm registry.
• Adds RRQ residual export format (auto_round:rrq) and inference-time composition (load_rrq_model, RRQLinear, precision switching utilities).
• Updates quantized linear kernels to expose _dequantize() for reuse, plus adds extensive CPU unit tests and user scripts; updates README(+CN) and ignores rrq_output/.

File summaries

File	Description
test/unit/test_cpu/algorithms/test_rrq.py	Comprehensive CPU unit tests for RRQ config, packing, reconstruction, inference switching, and Phase 2/3 behaviors.
test_rrq_qwen3_06b.py	Standalone script to quantize Qwen3-0.6B with RRQ and verify base/residual layout + (optional) load/forward.
test_rrq_lm_eval.py	Standalone script to run lm-eval across RRQ bit-widths (base+residual).
README.md	Adds RRQ announcement to “What’s New”.
README_CN.md	Chinese counterpart update for the RRQ “What’s New” entry.
auto_round/utils/common.py	Adds auto_round:rrq to supported formats list.
auto_round/inference/rrq_model.py	New loader that merges base + residual artifacts into an RRQ-enabled model by replacing layers with RRQLinear.
auto_round/inference/rrq_linear.py	New RRQLinear module and helpers to set uniform or random mixed precision across layers.
auto_round/inference/backend.py	Adds RRQ format constant (RRQ_FORMAT).
auto_round/export/formats/backends/rrq.py	New OutputFormat backend for auto_round:rrq residual export.
auto_round/export/formats/backends/__init__.py	Exposes RRQFormat in backend imports/exports.
auto_round/export/export_to_mlx/export.py	Fail-fast guard rejecting RRQ residual models for MLX export.
auto_round/export/export_to_gguf/conversion/base.py	Fail-fast guard rejecting RRQ residual models for GGUF export.
auto_round/export/export_to_autoround/export_to_rrq.py	Implements RRQ residual serialization + Phase 2 residual generation from base+raw weights.
auto_round/compressors/model_free.py	Frees packed shard tensors earlier to improve memory reclamation; tweaks a log message.
auto_round/cli/algorithms.py	Improves CLI arg merge logic to avoid mismatching boolean optional arguments with shared dest.
auto_round/autoround.py	Forces RRQ to route through calibrated path (disables model-free path) to avoid dropping residual planes.
auto_round/algorithms/registry.py	Registers RRQ config/quantizer modules and adds rrq to built-in algorithm order.
auto_round/algorithms/quantization/rrq/quantizer.py	Core RRQ quantizers (RTN multi-plane + SignRound per-plane tuning with frozen prefix) and packing logic.
auto_round/algorithms/quantization/rrq/config.py	RRQConfig implementation (fixed INT2 planes, tuning params, calibration requirement).
auto_round/algorithms/quantization/rrq/__init__.py	RRQ module exports.
auto_round/__init__.py	Exposes RRQConfig and lazily exports load_rrq_model / generate_rrq_residual.
auto_round_extension/torch/qlinear_torch.py	Fixes device usage in packing; adds _dequantize() helper and adjusts g_idx logic.
auto_round_extension/torch/qlinear_torch_zp.py	Adds _dequantize() helper and adjusts g_idx logic (symmetric/GPTQ-style).
.gitignore	Ignores rrq_output/ directory.

Review details

• Files reviewed: 24/25 changed files
• Comments generated: 7
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[a32543254]

LGTM

[AutoRoundBot]

CI Failure Analysis Report (Unit-Test-AutoRound)

• Commit: e4742da
• Run: https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=75454
• Totals: 1 failed test(s), 1 cluster(s) (0 known / 1 new)

New Issues (1)

No.1 — 1 occurrence(s)

🔍 AssertionError: /auto-round/test/unit/test_cpu/algorithms/test_rrq.py:977 has unclassified reason: Time-consuming lm_eval accuracy check; covered by nightly

📝 Basic info

• Affected tests (1): test_skip_ci_markers_have_one_classified_reason
• Logs (1): unittest_test_common_test_skip_ci_policy.log

🖥️ Log excerpt

unit/common/test_skip_ci_policy.py:58: in test_skip_ci_markers_have_one_classified_reason
    assert not violations, "\n".join(violations)
E   AssertionError: /auto-round/test/unit/test_cpu/algorithms/test_rrq.py:977 has unclassified reason: Time-consuming lm_eval accuracy check; covered by nightly
E   assert not ['/auto-round/test/unit/test_cpu/algorithms/test_rrq.py:977 has unclassified reason: Time-consuming lm_eval accuracy check; covered by nightly']

✨ AI analysis

• Category: Test Case Issue

• Confidence: high

• Root cause: The new RRQ accuracy test uses a skip_ci reason without one of the required classification prefixes, so the skip-policy test rejects it.

• Suggested fix: Prefix the reason with the Accuracy category.

• Patch:

  --- a/test/unit/test_cpu/algorithms/test_rrq.py
  +++ b/test/unit/test_cpu/algorithms/test_rrq.py
  @@ -972,7 +972,7 @@ class TestRRQAccuracy:
  -    @pytest.mark.skip_ci(reason="Time-consuming lm_eval accuracy check; covered by nightly")
  +    @pytest.mark.skip_ci(reason="Accuracy: Time-consuming lm_eval accuracy check; covered by nightly")

Notes

• Only top-3 issues receive AI analysis.
• To request an additional fix from Copilot, use "Quote reply" on the PR comment and @mention Copilot.

[wenhuach21]

As RRQ is currently difficult to deploy, I’d prefer not to expose this algorithm to users directly if possible. Instead, we can automatically switch to RRQ when a specific format is specified.

[wenhuach21]

Typically, we don't change the quantization method; we change the format instead if we want it to be adopted quickly by the community, following the successful adoption of AutoRound.