[jmitrevs]

Description

This combines #1476 and #1494. Also, due to many merge conflicts, it uses the renamed Altera backend of #1536, so it should only be merged after.

The legacy softmax was prone to overflow, so this PR made it use an accumulator type. Also, for stable and latency, the default types are better chosen.

For Altera, table generation logic was updated to be per-layer, not global, and copied from the configuration Memory attributes were added to enable memory banking on the FPGA, allowing for more efficient memory access.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change which adds functionality)

Tests

This updates test_softmax.py because previously it set attributes that had no effect.

We need to add Altera to the CI. Otherwise, other tests should not fail. However, Quartus tests were not a focus since that beckend will be retired with #1537. (It may introduce merge conflicts, but those should be fairly easy to solve.)

Checklist

• I have read the guidelines for contributing.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have installed and run pre-commit on the files I edited or added.
• I have added tests that prove my fix is effective or that my feature works.

[JanFSchulte]

Detailed review of the current state of the softmax changes by Claude:

Review at 68aa4e6

Thanks for folding #1552 in, and for 68aa4e6. With the latency inverse table addressed by inv_inp_t, HGQ2 QSoftmax is bit-exact again in every latency configuration I tried. The new inp_norm_t inference fixes the stable resolution problem behind #1512 when the table types are auto: 822 → 7 argmax flips per 2000 inputs at ap_fixed<24,12>, two classes.

Three things I'd like fixed before merging:

1. Heterogeneous HGQ quantizers in io_stream now give silently wrong results. Main refuses these with NotImplementedError. This branch converts them, and an HGQ2 QDense → ReLU → QDense model gives 2000/2000 rows different from Keras (max |diff| 1.92). This affects every FPGA backend, not just softmax.
2. The default config_from_keras_model path regresses. With granularity='model' (the default), the softmax table types fall back to the model precision instead of being inferred. At ap_fixed<24,12> every output is 0.5 (999/2000 argmax flips; main: 822). At the default fixed<16,6> with ten classes, stable flips go from 31 to 85.
3. Stable HGQ tables are now capped at TableSize. Main never caps them. With 12-bit table inputs, 933/2000 outputs differ from Keras and 8 argmax decisions change.

With the suggested changes for (2) and (3) applied, those stable cases drop to 7 and 21 flips (main: 822 and 31), all the softmax HGQ cases below are bit-exact, and the Vitis subset of test_softmax.py still passes.

1. Heterogeneous quantizers in io_stream

hls4ml/hls4ml/backends/fpga/passes/hgq_proxy_model.py

Lines 74 to 83 in 68aa4e6

	# instead of error assertion just force homogeneous quantisation
	if model.config.config['IOType'] != 'io_parallel':
	k, b, i = node.mask_kbi
	k_val = int(np.max(k))
	b_val = int(np.max(b))
	i_val = int(np.max(i))
	inp_var = node.get_input_variable()
	inp_var.type.precision = FixedPrecisionType(b_val, i_val, bool(k_val))
	model.remove_node(node)
	return True

This used to raise NotImplementedError for heterogeneous quantizers outside io_parallel. Now it collapses them to one type and carries on, which turns a clear error into wrong output. An HGQ2 QDense(16) → ReLU → QDense(4) model with the default per-channel activation quantizers gives 2000/2000 rows different from Keras in io_stream (max |diff| 1.92). Main refuses to convert it.

The collapse also loses information. The width is max(b) with max(i) integer bits, so channels that need fractional bits lose them; covering every channel takes at least max(i) + max(b - i) bits. The new type also drops the quantizer's rounding and saturation modes. Even with both fixed, one type can't reproduce per-channel quantization, so this can't be bit-exact.

I'd keep the error, and move any io_stream relaxation to its own PR since it touches every FPGA backend. Suggested change for L74–83:

        if model.config.config['IOType'] != 'io_parallel':
            raise NotImplementedError('Heterogenous quantization for activations is only supported with IOType=io_parallel')

2. Model-level configs skip type inference

hls4ml/hls4ml/model/layers.py

Lines 1058 to 1065 in 68aa4e6

	# A standalone Keras Softmax layer arrives as 'Softmax'
	self.set_attr('activation', self.get_attr('activation').lower())
	# set the needed types if needed
	self._set_type_t('exp_table')
	self._set_type_t('inv_table')
	self._set_type_t('inv_inp')
	self._set_type_t('inp_norm')

With config_from_keras_model(...) at its default granularity='model', none of these types is configured per layer or per type. get_precision resolves them to model_default_t, so infer_precision never treats them as auto, and the tables and address types use the model precision. At ap_fixed<24,12> every output is 0.5 (999/2000 argmax flips; main: 822, two classes). At the default fixed<16,6> with ten classes, stable flips go from 31 to 85.

Treating types that only resolve to the model default as auto fixes it in my runs (999 → 7 and 85 → 21 flips). Every HGQ case stays bit-exact, and the Vitis subset of test_softmax.py still passes. Suggested change for L1058–1061:

        # A standalone Keras Softmax layer arrives as 'Softmax'
        self.set_attr('activation', self.get_attr('activation').lower())

        # Internal types that only resolve to the model-wide default are inferred instead
        if not self.get_attr('_bit_exact', False):
            for name in ('exp_table', 'inv_table', 'inv_inp', 'inp_norm', 'accum'):
                current = self.get_attr(f'{name}_t')
                if current is not None and current.name != 'model_default_t':
                    continue
                if self.model.config.get_precision(self, name)[1] == 'model_default_t':
                    self.set_attr(f'{name}_t', NamedType(f'{self.name}_{name}_t', UnspecifiedPrecisionType()))

        # set the needed types if needed

3. TableSize cap on bit-exact tables

hls4ml/hls4ml/backends/fpga/passes/fix_softmax_table_size.py

Lines 25 to 28 in 68aa4e6

	if node.get_attr('_bit_exact', False):
	# Bit-exact (HGQ) tables are sized to address every input value, so bound them by table_size
	exp_table_size = min(table_size, exp_table_size)
	inv_table_size = min(table_size, inv_table_size)

This caps bit-exact (HGQ) stable tables at TableSize, which main never did. bit_exact sizes them to 2^width of their address type, so capping drops low address bits. With exp_iq_conf/inv_iq_conf at 9 fractional bits (13- and 12-bit table inputs), the tables shrink from 8192 and 4096 entries to 1024. Then 933/2000 outputs differ from Keras, 8 of them with a different argmax. On Vivado this happens silently; only Altera's validate_hgq_softmax_types warns.

If the aim is to bound table memory, I'd keep bit-exact tables full size and warn, so the user decides. My #1531 sizes them to exactly 2^width, so settling this here unblocks that PR too. The change below makes the case above bit-exact again and keeps the tests passing. It needs import warnings at the top, and the test_softmax_explicit_table_size docstring would need updating. Suggested change for L25–28:

            if node.get_attr('_bit_exact', False) and max(exp_table_size, inv_table_size) > table_size:
                # Bit-exact (HGQ) tables address every value of their input type; shrinking them breaks bit-exactness
                warnings.warn(
                    f'Softmax layer {node.name}: bit-exact tables ({exp_table_size}, {inv_table_size} entries) '
                    f'exceed TableSize={table_size}; keeping them to stay bit-exact.',
                    stacklevel=1,
                )

Smaller items

inv_inp integer-bit override

hls4ml/hls4ml/model/layers.py

Lines 209 to 212 in 68aa4e6

	# We skip if table is quantised by HGQ (_bit_exact property) since we want to match Keras and HGQ saturation.
	# Only the stable implementation bounds the sum of exponentials by the number of inputs.
	if name == 'inv_inp' and self.get_attr('implementation') == 'stable' and not self.get_attr('_bit_exact', False):
	self.get_attr(f'{name}_t').precision.integer = 1 + math.ceil(math.log2(max(self.get_input_variable().shape)))

This now only lands when the user sets inv_inp explicitly, and then it silently rewrites their type: ufixed<18,8,RND_CONV,SAT> with 8 inputs becomes ufixed<18,4,RND_CONV,SAT>. Under the default model-level config it is lost, because the type keeps the shared model_default_t name, which is emitted as the model default. With auto, the inference replaces it. max(shape) also overestimates for multidim inputs, where the sum is bounded by n_slice. Could this move into _infer_softmax_precision for stable, using n_slice, so it applies exactly when the type is inferred?

1-bit ac_fixed widening

hls4ml/hls4ml/backends/fpga/passes/hgq_proxy_model.py

Lines 16 to 33 in 68aa4e6

	def to_acfixed(k, b, i, RND, SAT):
	k = 'false' if k == 0 else 'true'
	if b == 1:
	# Currently Altera ac_fixed requires at least two bits for both signed and unsigned cases
	# Should be fixed in the future once Altera supports 1-bit unsigned ac_fixed
	if k == 1:
	print(
	f'Warning: Current variable is 1-bit signed (ac_fixed<1,{i},{k}>)'
	'Current Altera HLS backend does not support 1 bit types therefore conversion '
	'of signed 1 bit type to 2 bits will be lossy when widened to 2 bits.'
	)
	# Widen by assigning the extra bit as the sign bit
	b = 2
	i += 1
	k = 1
	return f'ac_fixed<{b},{i},{k},AC_{RND},AC_{SAT}>'

(The same widening is in fpga_types.py, L99–113.)

• In to_acfixed, k is already the string 'true'/'false', so if k == 1 never fires, and k = 1 then emits 1 as the signedness argument.
• Making the type signed changes which values survive quantization. Checked with the bundled ac_fixed.h under g++: ac_fixed<1,1,false,AC_RND,AC_SAT> maps −0.7 and −1.7 to 0, while the widened ac_fixed<2,2,true,AC_RND,AC_SAT> maps them to −1 and −2. For a signed 1-bit type, 0.7 → 0 becomes 0.7 → 1. So an unsigned 1-bit HGQ channel, for example one after a ReLU, can pass negative values on Altera.
• No 2-bit type reproduces 1-bit saturation exactly, so this needs an explicit clamp to the 1-bit range. Main's unsigned ac_fixed<2,I,false> at least keeps the sign. Also print → warnings.warn.

Leftover debug code

hls4ml/hls4ml/converters/keras_v3/hgq2/softmax.py

Lines 124 to 129 in 68aa4e6

	# For multidim implementation assume first input is the tensor we are operating on
	activation = 'softmax'
	# import pdb; pdb.set_trace()
	# if len(in_tensors[0].shape[1:]) > 1:
	# if (1 not in in_tensors[0].shape[1:]) or (len(in_tensors[0].shape[1:]) > 2):
	# activation = 'softmax_multidim'

A pdb line and a dead block. Vivado picks softmax_multidim from n_outer/n_inner in the function template, and Altera io_parallel rejects multidim inputs, so these lines can just be activation = 'softmax'.

Shared template mutated in place

hls4ml/hls4ml/backends/altera/passes/core_templates.py

Lines 277 to 280 in 68aa4e6

	if params['implementation'] == 'stable':
	self.template = softmax_config_template + softmax_config_table_template_stable
	return self.template.format(**params)

This assigns to the shared template instance. After the first stable softmax, every later softmax in the process (any implementation, any model) is formatted with the stable template. That's harmless today because the stable template is a superset, but it's easy to avoid. Suggested change for L277–280:

            if params['implementation'] == 'stable':
                return (softmax_config_template + softmax_config_table_template_stable).format(**params)

        return self.template.format(**params)

Nits

• implemenation → implementation in fix_softmax_table_size.py L22.
• test_softmax.py L188: the comment is stale after 68aa4e6. The inverse table is now addressed by inv_inp_t; it's inferred as a copy of accum_t, so the size is still right.

Not in the diff

• Vivado's SoftmaxConfigTemplate still has main's fallbacks: it derives accum_t when that is model_default_t, swaps inv_inp_t for exp_table_t, and keeps the inp_norm_t branch with its exp_table_size clamp. These are now dead or compete with _infer_softmax_precision. Worth removing here so there's one source of truth.
• Altera's init_softmax still asserts 1-D input for io_parallel, so the new io_parallel softmax_multidim kernel can't be reached. Planned for a follow-up?
• Once (2) is fixed, the inferred latency defaults become the default path, and they're less accurate than main's at fixed<16,6>: mean abs error 0.083 vs 0.054 (two classes) and 0.018 vs 0.012 (ten classes). The 1024-entry inverse table is addressed by ufixed<19,9>, a step of 0.5 in the sum. Not blocking, but a narrower saturating inv_inp_t for latency would avoid a quiet accuracy drop.
• Tests that would have caught (1)–(3): a model-level-only config at a wide precision, heterogeneous HGQ activations in io_stream (expecting the error), and an HGQ QSoftmax with table inputs wider than 10 bits.

What I ran

At 68aa4e6, on Linux with g++ csim and no vendor tools:

• test_softmax.py -k "Vitis or (explicit_table_size and Catapult)": 102 passed, including the 10 new tests. Altera and XLS not run.
• HGQ2 0.2.0 with Keras 3.15.1 on JAX, compared bit for bit with Keras on 2000 inputs:

Configuration	main	this PR
QDense → QSoftmax, default quantizers: stable and latency × io_parallel and io_stream × 8 and 32 classes	0 rows differ	0
Latency, inv_inp with 2 fractional bits	0	0
Latency, saturating inv_inp	0	0
Stable, 12-bit table inputs	0	933
QDense → ReLU → QDense, heterogeneous activations, io_stream	NotImplementedError	2000

• The flip counts come from plain Input → Softmax graphs on 2000 Gaussian logits (σ = 1), compared with float softmax.

[jmitrevs]

I tried to address the review comments. Let's see how the pytests go. I am ignoring Quartus failures since that backend will go away.

[JanFSchulte]

n_outer and n_inner are only set by the keras frontends, so we should add that to the torch and onnx ones, otherwise multidimensional softmax with io_stream will be rejected for those frontends.

[jmitrevs]

Should be fixed now.

[JanFSchulte]

Latency softmax: why the inferred types lose to main at small logits

A correction to my earlier review first: I wrote that latency with inferred types is less accurate than main's fixed defaults. That only holds for small logits. Across logit scales it's a trade-off, and the loss traces to one thing: the inverse table isn't resized along with the inferred inv_inp_t.

Mechanism

• Range. For latency, accum_t gets the exp table's 8 integer bits plus ⌈log₂ n⌉, and inv_inp_t copies it: ufixed<19,9> at 2 classes, ufixed<22,12> at 10. That is the tight overflow-free range, since a latency exponential can reach 256. (Stable gets a tighter bound because normalized exponentials are ≤ 1.)
• Resolution. The inverse table is indexed by the top log₂(InvTableSize) = 10 bits of inv_inp_t, and fix_softmax_table_size only ever shrinks the table. So every integer bit added for range halves the lookup resolution: the address step in the sum is 2^(8 + ⌈log₂ n⌉ − 10), which is 0.5 at 2 classes and 4 at 10. With logits of order 1 the sum is about n, so 1/sum is off by up to 12–25%.
• Main uses a fixed ap_fixed<18,8>, a 0.25 step on [0, 128) whatever n is. But its accum_t is the model default (ap_fixed<16,6>, WRAP), which overflows once an exponential or the sum passes 32. Setting main's accum_t to ufixed<19,9> alone cuts its σ = 2 error at 10 classes from 0.41 to 0.006.

Swapping only inv_inp_t back to main's type on this branch restores main's small-logit accuracy (10 classes, σ = 1: 0.018 → 0.0016), so the regression comes from that one type.

Measured

Mean absolute error vs float softmax, latency, fixed<16,6> input, 2000 Gaussian logits per σ, Vitis csim at 02c2884:

	n=2, σ=0.5	σ=1	σ=2	σ=4	n=10, σ=0.5	σ=1	σ=2	σ=4
main	0.035	0.054	0.467	1.275	0.0017	0.0125	0.407	0.606
this PR (inferred)	0.081	0.083	0.075	0.093	0.0250	0.0182	0.0102	0.0290
this PR + main's inv_inp_t	0.035	0.037	0.050	0.181	0.0018	0.0016	0.0093	0.176
A: inverse table grown to 2^(I+2)	0.035	0.037	0.044	0.078	0.0017	0.0015	0.0016	0.0263
B: inv_inp_t = ufixed<17,7,RND,SAT>	0.017	0.019	0.037	0.174	0.0015	0.0014	0.0092	0.176

So this branch is worse than main only at σ ≤ 1, and 5–40× better at σ ≥ 2.

Two possible fixes

• A: grow the inverse table with the inferred type. Use 2^(I+2) entries, where I is the integer bits of inv_inp_t; this keeps main's 0.25 step (2048 entries at 2 classes, 16384 at 10). It matches or beats main at every σ and class count above. The cost is memory, roughly 1024 · 2^⌈log₂ n⌉ entries, so it needs a cap, and it should only apply when inv_inp_t is inferred and InvTableSize isn't set explicitly.
• B: cap inv_inp_t at 7 integer bits with saturation. No extra memory, and the most accurate at small logits (2–3× better than main at 2 classes), but sums above 128 saturate, so σ = 4 degrades to about 0.17 (still well below main).

My preference is A with a cap, falling back to B's saturating type above the cap. Happy to go with B if table memory is the bigger concern.

With wide inputs such as ap_fixed<24,12>, neither fix matters much. The latency exp table is addressed by the raw input's top 10 bits, a step of 4 in logit space, and that dominates on every branch (487/2000 argmax flips at 2 classes on main and here). Only the stable implementation avoids it.

[jmitrevs]

For the Latency softmax, I modified what is inferred by auto for the precision--explicit precision are not modified. So I chose option 2 for when auto inference is used. A user can explicitly grow the table size and set the precision if they desire. This seems a cleaner solution for the default.

[gksks1029]

I also encountered a significant accuracy drop after softmax while working through the Xilinx/Vitis flow with a pretrained MNIST LeNet-1 model.

Using the Vitis backend with io_stream, HLS C++ inference accuracy on the first 100 MNIST test images dropped from 100% in Keras to 8% on main (863a4e972). Removing softmax restored accuracy to 100%.

I tested this PR (92dfc9e34), and accuracy with softmax returned to 100% on the same samples. This comparison used CPU C++ inference, not FPGA execution.

This summary were prepared with assistance from Codex (GPT-6.1 Sol).

[Datayx]

Hi, thanks for hls4ml! Some numbers that might help with this PR.

Setup: hls4ml 1.3.0, Vitis, io_parallel, default softmax_stable, 4-bit QKeras MLP (15→32→32→5), jet-tagging dataset, 10 seeds.
With default types the softmax outputs don't sum to 1 (mean 1.08, max 1.249). Argmax is mostly fine, but ECE goes from 0.009 (Keras) to 0.057. Renormalising the outputs fixes it.

The cause seems to be inv_inp_t = fixed<18,8>: the 1/x table is indexed by its top 10 bits, which leaves 0.25-wide bins over a sum in [1, 5].
Setting inv_inp_t = ufixed<18,3,RND,SAT> brings the sums to 1.000 and ECE back to Keras level averaged over 10 seeds (0.0093 vs 0.0092), with no change in latency or resources (16,070 vs 16,114 LUTs post-route).

Note: in 1.3.0 setting this via the config is ignored. Editing the node before compile() works:

from hls4ml.model.types import RoundingMode, SaturationMode
p = hls_model.graph[softmax_name].get_attr('inv_inp_t').precision
p.width, p.integer, p.signed = 18, 3, False
p.rounding_mode, p.saturation_mode = RoundingMode.RND, SaturationMode.SAT
hls_model.compile()

Details: https://github.com/Datayx/qnn-fpga. Happy to rerun on the #1542 branch.