QKeras recurrent layers (QSimpleRNN, QLSTM, QGRU): weight quantizers are not applied, so results differ from QKeras (Keras 2 and Keras 3) #1579

Summary

QKeras recurrent layers (QSimpleRNN, QLSTM, QGRU) quantize their weights, and QKeras computes with the quantized values. hls4ml reads these quantizers from the model but never applies them to the weights of its SimpleRNN, LSTM and GRU layers. hls4ml then computes with slightly different weights than QKeras, so its output differs from the QKeras model.

This affects both QKeras packages hls4ml supports:

• qkeras (QKeras 0.9, for Keras 2);
• qkeras-v3 (QKeras for Keras 3).

They go through the same hls4ml code, so one fix covers both.

With 8-bit weights (quantized_bits(8, 0, alpha=1)), the largest difference from QKeras on the Vivado backend, using the script below, is:

"Fixed" means with the change described under "What to change". The QSimpleRNN numbers need #1577 on Vivado and Vitis, which adds SimpleRNN to those backends. On main, QSimpleRNN on Vivado and Vitis converts to an empty design.

Background

A QKeras recurrent layer takes three weight quantizers, for example:

QLSTM(8, kernel_quantizer='quantized_bits(8, 0, alpha=1)',
         recurrent_quantizer='quantized_bits(8, 0, alpha=1)',
         bias_quantizer='quantized_bits(8, 0, alpha=1)')

The model stores the weights as floats, and QKeras applies the quantizers each time it computes.

How hls4ml reads them. hls4ml's QKeras parser for recurrent layers, parse_qrnn_layer in hls4ml/converters/keras/qkeras.py, reads the quantizers into three layer attributes: weight_quantizer, recurrent_weight_quantizer and bias_quantizer.

• Keras 2 models use it directly.
• Keras 3 models (QKeras v3) get there too. hls4ml's Keras 3 converter has QKeras handlers only for QDense, the convolutions and QConv2DBatchnorm (hls4ml/converters/keras_v3/qkeras/layer.py). Recurrent layers fall back to the Keras 2 handler, and conversion prints Keras v2 handler used for layer ....

Where the quantizers should be applied. For other layers, the quantizer is applied when the weights are added to the layer. For example, Dense.initialize in hls4ml/model/layers.py does:

self.add_weights(quantizer=self.get_attr('weight_quantizer'), compression=...)
self.add_bias(quantizer=self.get_attr('bias_quantizer'))

SimpleRNN.initialize, LSTM.initialize and GRU.initialize in the same file add their weights without a quantizer:

self.add_weights()
self.add_weights_variable(name='recurrent_weight', var_name='wr{index}')
self.add_weights_variable(name='bias', var_name='b{index}')

So hls4ml takes the float weights and converts them to the weight type (fixed<8,1> here) by truncation, while QKeras rounds them. The two sets of weights differ by up to one step of the type.

How to reproduce

The same script works with both packages. Run it with either:

• qkeras (QKeras 0.9) and TensorFlow 2.14 with Keras 2, the versions hls4ml's Keras 2 tests use; or
• qkeras-v3 with Keras 3, which is what pip install hls4ml[qkeras-v3] installs.

qkeras-v3 pins its own Keras and TensorFlow versions, so give it an environment of its own.

QKeras 0.9 with TensorFlow 2.14 fails inside QKeras's recurrent layers with AttributeError: module 'tensorflow.python.util.nest' has no attribute 'is_sequence', and QGRU then also fails on array_ops.unstack. QKeras calls two TensorFlow functions that TensorFlow 2.14 removed. The first lines of the script put them back when they're missing. They aren't part of the fix, and QKeras v3 doesn't need them.

import keras
import numpy as np
import tensorflow as tf
from tensorflow.python.ops import array_ops
from tensorflow.python.util import nest

# QKeras 0.9.0 (Keras 2) calls two TensorFlow functions that TensorFlow 2.14 removed. Not needed for QKeras v3.
if not hasattr(nest, 'is_sequence'):
    nest.is_sequence = nest.is_nested
if not hasattr(array_ops, 'unstack'):
    array_ops.unstack = tf.unstack

from qkeras import QGRU, QLSTM, QSimpleRNN  # noqa: E402
from qkeras.quantizers import quantized_bits  # noqa: E402

import hls4ml  # noqa: E402

q = 'quantized_bits(8, 0, alpha=1)'
quantizers = dict(kernel_quantizer=q, recurrent_quantizer=q, bias_quantizer=q)
layers = {
    'QSimpleRNN': lambda: QSimpleRNN(8, activation='relu', **quantizers),
    'QLSTM': lambda: QLSTM(8, activation='tanh', recurrent_activation='sigmoid', **quantizers),
    'QGRU': lambda: QGRU(8, activation='tanh', recurrent_activation='sigmoid', reset_after=True, **quantizers),
}
X = np.round(np.random.default_rng(0).uniform(-0.5, 0.5, (50, 6, 4)) * 2**7) * 2**-7

for name, make_layer in layers.items():
    keras.utils.set_random_seed(0)
    x_in = keras.Input((6, 4))
    model = keras.Model(x_in, make_layer()(x_in))
    y_qkeras = model.predict(X, verbose=0)

    config = hls4ml.utils.config_from_keras_model(
        model, granularity='name', backend='Vivado', default_precision='ap_fixed<24,10>'
    )
    hls_model = hls4ml.converters.convert_from_keras_model(
        model, hls_config=config, backend='Vivado', output_dir=f'hls4mlprj_{name}'
    )
    hls_model.compile()
    y_hls = hls_model.predict(X).reshape(y_qkeras.shape)

    rnn = model.layers[1]
    kernel = rnn.get_weights()[0]
    hls_kernel = hls_model.graph[rnn.name].get_weights('weight').data.reshape(kernel.shape)
    quantized = np.array_equal(hls_kernel, np.asarray(quantized_bits(8, 0, alpha=1)(kernel)))
    print(f'{name}: kernel quantized like QKeras: {quantized}, max difference from QKeras: {np.max(np.abs(y_hls - y_qkeras)):.4f}')

The script looks the layer up by its actual name, because QKeras v3's recurrent layers don't keep a name= passed to them.

Output today with QKeras 0.9, with #1577 so that QSimpleRNN works on Vivado:

QSimpleRNN: kernel quantized like QKeras: False, max difference from QKeras: 0.0610
QLSTM: kernel quantized like QKeras: False, max difference from QKeras: 0.0968
QGRU: kernel quantized like QKeras: False, max difference from QKeras: 0.0146

With QKeras v3 the first column is the same, and the differences are 0.0610, 0.1054 and 0.0231.

No HLS tool is needed: compile() builds the C++ with the system compiler.

What to change

1. In hls4ml/model/layers.py, pass the quantizers when SimpleRNN, LSTM and GRU add their weights:

   • self.add_weights(quantizer=self.get_attr('weight_quantizer'));
   • quantizer=self.get_attr('recurrent_weight_quantizer') for recurrent_weight;
   • quantizer=self.get_attr('bias_quantizer') for bias. GRU with reset_after=True also has a recurrent_bias, which Keras stores in the same bias array, so check whether QKeras applies bias_quantizer to it too.

   For layers that don't come from QKeras these attributes are None, and nothing changes.

   With this change, the script prints True for all three layers with both packages, and the differences in the "fixed" columns above.

   • QKeras 0.9: the remaining difference comes from the lookup tables used for tanh and sigmoid.
   • QKeras v3: QLSTM and QGRU keep a somewhat larger difference (0.018 and 0.019). That's worth a look while working on this.

2. Check the Altera backend. Its init_lstm, in hls4ml/backends/altera/altera_backend.py, splits the LSTM weights per gate and quantizes the pieces itself. After step 1, the weights arrive already quantized. Quantizing again with the same quantizer changes nothing, but init_lstm uses weight_quantizer for the kernel, the recurrent kernel and the bias. It should use recurrent_weight_quantizer and bias_quantizer for the last two, and this matters when the three quantizers differ. The Quartus backend has the same code, but Retire Quartus backend #1537 removes that backend, so leave it as it is.

3. Tests. The QKeras recurrent tests exist twice: in test/pytest/test_qkeras.py for QKeras 0.9, and in test/pytest/test_qkerasV3.py for QKeras v3. CI runs the second file in a separate job with pip install .[qkeras-v3].

   • test_qsimplernn, test_qlstm and test_qgru in both files compare with QKeras with atol=0.1, which is loose enough to hide this problem. Tighten it to a value the fixed code passes with a margin, for example 0.02 or 0.03. Check the actual differences with both packages first.
   • Once Fix recurrent layers on Vivado and Vitis, and add SimpleRNN #1577 is merged, add 'Vivado' and 'Vitis' to test_qsimplernn in both files, and add 'Vitis' to test_qlstm and test_qgru. In test_qkeras.py, do the same in the QSimpleRNN, QLSTM and QGRU entries of NO_QUANT_LAYERS.
   • A test that would fail today: a QLSTM or QSimpleRNN with 8-bit weight quantizers, compared with QKeras at the tighter tolerance.

Related problems, not part of this issue

These came up while looking at QKeras recurrent layers. Each could be its own issue.

• State quantizer: QKeras's state_quantizer is parsed into accum_quantizer by parse_qrnn_layer and never used.
• Activation quantizers in LSTM and GRU: QKeras applies them inside the cell. hls4ml only applies them where the layer's result type is written, so LSTM and GRU with quantized activations are approximated. SimpleRNN is exact, because its state is its output.
• hard_sigmoid and hard_tanh: these aren't supported by recurrent layers on Vivado and Vitis. QKeras's quantized_sigmoid and quantized_tanh, and QLSTM's default recurrent activation, map to them. After Fix recurrent layers on Vivado and Vitis, and add SimpleRNN #1577 the conversion stops with a clear message; before, it failed in the C++ compiler. nnet_activation.h already has both functions for standalone activations, so nnet_recr_activations.h would need wrappers for them, plus their slope and shift settings.
• QBidirectional (QKeras v3): not supported. The conversion stops with ValueError: Layer <class 'qkeras.qrecurrent.QBidirectional'> not found in registry.