Altera: RecurrentReuseFactor has no effect #1580

Summary

On the Altera backend, the RecurrentReuseFactor setting of SimpleRNN, LSTM and GRU layers has no effect. The backend offers the setting and writes it into every config that config_from_keras_model generates. Its layer setup then overwrites it with the layer's ReuseFactor, so whatever the user sets is silently ignored.

#1577 found and removed the same setting on the Vivado and Vitis backends. This issue is about doing the same for Altera, so that all backends behave the same way. The Quartus backend has the same problem, but #1537 removes that backend, so it isn't covered here.

Background

A recurrent layer does two matrix multiplications in every time step: the input with the kernel, and the previous state with the recurrent kernel. The reuse factor sets how many clock cycles a multiplier is reused for, which trades multipliers for latency. ReuseFactor is meant for the first multiplication and RecurrentReuseFactor for the second.

On Vivado and Vitis, #1577 measured that the two multiplications run side by side in each time step. The step takes as long as the slower of the two, so a different reuse factor for one of them can only waste multipliers. For example, an LSTM with input reuse factor 3 and recurrent reuse factor 64 took 641 cycles and 192 DSPs. With 48 and 64 it took the same 641 cycles and only 72 DSPs. #1577 therefore removed RecurrentReuseFactor there, uses ReuseFactor for both multiplications, and warns when a config still sets it.

How to reproduce

import keras

import hls4ml

model = keras.Sequential([keras.Input((8, 6)), keras.layers.LSTM(16, name='rnn')])

config = hls4ml.utils.config_from_keras_model(model, granularity='name', backend='Altera')
print('Generated config:', {k: v for k, v in config['LayerName']['rnn'].items() if 'Reuse' in k})
config['LayerName']['rnn']['ReuseFactor'] = 8
config['LayerName']['rnn']['RecurrentReuseFactor'] = 4
hls_model = hls4ml.converters.convert_from_keras_model(
    model, hls_config=config, backend='Altera', output_dir='hls4mlprj_rrf_Altera'
)
print('recurrent_reuse_factor used:', hls_model.graph['rnn'].get_attr('recurrent_reuse_factor'))

Output:

Generated config: {'ReuseFactor': 1, 'RecurrentReuseFactor': 1}
recurrent_reuse_factor used: 8

RecurrentReuseFactor was set to 4, but the layer uses 8. The script only converts the model, so no HLS tool or compiler is needed.

Where in the code

• Settings, hls4ml/backends/altera/altera_backend.py:
  • _register_layer_attributes registers recurrent_reuse_factor as a ConfigurableAttribute for SimpleRNN, LSTM and GRU;
  • init_gru, init_lstm and init_simple_rnn each start with layer.set_attr('recurrent_reuse_factor', reuse_factor), which overwrites the user's value.
• Where the value is used: the GRU template in hls4ml/backends/altera/passes/recurrent_templates.py reads recurrent_reuse_factor for the recurrent multiplication, and init_gru and init_lstm adjust it with set_closest_reuse_factor. The internal attribute must keep being set; only the user-facing setting is the problem.

What to change

Recommended, to match #1577:

1. Remove the ConfigurableAttribute('recurrent_reuse_factor', ...) registration. config_from_keras_model then stops writing RecurrentReuseFactor into generated configs.
2. Keep setting the internal recurrent_reuse_factor from ReuseFactor in the init_* functions, as now.
3. Warn when a config still sets RecurrentReuseFactor. Since the setting is no longer registered, it only appears in layer.attributes if the user's config put it there. _warn_removed_reuse_factors in hls4ml/backends/vivado/vivado_backend.py, added by Fix recurrent layers on Vivado and Vitis, and add SimpleRNN #1577, does exactly this and can be reused or copied.
4. Regenerate the attribute reference, docs/ir/attributes.rst, with docs/attr_doc_gen.py. Don't edit it by hand.

The alternative is to make the setting take effect. That's only worth it if the two multiplications run one after the other on Altera, so that a different reuse factor for each saves resources without adding latency. Showing that needs synthesis with the Altera tools, so it's a bigger task than the recommended change.

How to test

• Add a test in test/pytest/test_rnn.py that converts a SimpleRNN, LSTM and GRU on Altera and checks:

  • config_from_keras_model no longer writes RecurrentReuseFactor;
  • setting it gives a warning, and the layer's recurrent_reuse_factor equals its ReuseFactor.

  Conversion alone is enough for this, so no tools are needed.

• Run the existing recurrent tests for Altera: test_rnn.py and test_recurrent_pytorch.py. Tests that call compile() need the Altera HLS compiler (ahls). Without it they fail the same way before and after the change.

Note

The Catapult backend registers and overwrites the same setting, in hls4ml/backends/catapult/catapult_backend.py. It isn't included here, because converting the LSTM above with Catapult currently fails for an unrelated reason: UnboundLocalError: cannot access local variable 'out_0'. That's worth its own issue.