Add Conv1DTranspose and Conv2DTranspose to the Vivado and Vitis backends

docs/backend/vitis.rst

@@ -17,6 +17,19 @@ Known issues
   they use ``io_stream`` and the ``Latency`` strategy, or the ``Resource`` strategy with a reuse factor of 1. ``Vitis HLS`` 2024.1 is not affected. With
   2023.2, raising the clock uncertainty to 35% (``clock_uncertainty='35%'``) was enough for the models tested, at the cost of one to three extra
   clock cycles per time step.
+* With ``Vivado HLS`` 2020.1, ``Conv1DTranspose`` and ``Conv2DTranspose`` layers with the ``Latency`` strategy may not meet the target clock at the
+  default clock uncertainty. Raising the clock uncertainty to 27% (``clock_uncertainty='27%'``) was enough for most of the models tested, though some
+  layers required more. The same can happen with ``io_stream`` and the ``Resource`` strategy at a small reuse factor. ``Vitis HLS`` 2024.1 is not affected.
+* With ``Vitis HLS`` 2024.1, the ``io_stream`` implementation of ``Conv1DTranspose`` and ``Conv2DTranspose`` takes one clock cycle more per input
+  than the interval given in :ref:`Transposed convolution` in some layers where the first of its two processes is the slower one.
+* The following are not supported with ``Vitis HLS`` 2025.1:
+
+  * ``Conv1D`` and ``Conv2D`` with ``io_parallel``: co-simulation does not finish when the model is a dataflow region, which is the default for
+    models with convolutions.
+  * ``Dense`` with ``io_parallel`` and the ``Resource`` strategy: C synthesis did not finish in six hours for most of the models tested,
+    even small ones.
+  * ``Conv1DTranspose`` and ``Conv2DTranspose``.
+  * The ``Fused`` strategy, whose interval is less predictable with this version (see :doc:`Fused Strategy <../advanced/fused>`).
 
 Fused strategy
 ==============

docs/ir/attributes.rst

@@ -1383,6 +1383,238 @@ Backend-specific attributes
 
   * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Catapult, Coyote
 
+Conv1DTranspose
+===============
+Base attributes
+---------------
+* result_t: NamedType
+
+  * The datatype (precision) of the output tensor.
+
+* weight_t: NamedType
+
+* bias_t: NamedType
+
+Type attributes
+---------------
+* index: int
+
+  * Internal node counter used for bookkeeping and variable/tensor naming.
+
+* in_width: int
+
+* out_width: int
+
+* n_chan: int
+
+* n_filt: int
+
+* filt_width: int
+
+* stride_width: int
+
+* dilation_width: int (Default: 1)
+
+* crop_left: int
+
+Weight attributes
+-----------------
+* weight: WeightVariable
+
+* bias: WeightVariable
+
+Configurable attributes
+-----------------------
+* trace: int (Default: False)
+
+  * Enables saving of layer output (tracing) when using hls_model.predict(...) or hls_model.trace(...)
+
+* result_t: NamedType
+
+  * The datatype (precision) of the output tensor.
+
+* weight_t: NamedType
+
+* bias_t: NamedType
+
+Backend-specific attributes
+---------------------------
+* accum_t: NamedType
+
+  * The datatype (precision) used to store intermediate results of the computation within the layer.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Quartus, Catapult, SymbolicExpression, Altera, Coyote, Libero, XLS
+
+* reuse_factor: int (Default: 1)
+
+  * The number of times each multiplier is used by controlling the amount of pipelining/unrolling. A lower number results in more parallelism and lower latency at the expense of the resources used. Reuse factor = 1 corresponds to all multiplications executed in parallel, and hence, the lowest possible latency.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Quartus, Catapult, SymbolicExpression, Altera, Coyote, Libero, XLS
+
+* parallelization_factor: int (Default: 1)
+
+  * The number of outputs computed in parallel. Essentially the number of multiplications of input window with the convolution kernel occurring in parallel. A higher number results in more parallelism (lower latency and II) at the expense of resources used. Currently only supported in io_parallel.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* slot_groups: tuple (Default: ())
+
+  * Set by the backend. The output is computed in blocks of stride positions, one block for each input position. For each kernel element, in the order of the rows of the weight matrix, the position in the block that it adds to.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* slot_shifts: tuple (Default: ())
+
+  * Set by the backend. For each kernel element, in the order of the rows of the weight matrix, how many blocks before the current block its input is, as (height, width).
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* n_buf: int (Default: 0)
+
+  * Set by the backend. The number of inputs multiplied for each block of outputs, which is the number of kernel elements times the number of input channels.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* first_block_width: int (Default: 0)
+
+  * Set by the backend. The index of the first block of outputs along this axis that cropping does not remove entirely.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* n_blocks_width: int (Default: 0)
+
+  * Set by the backend. The number of blocks of outputs along this axis that cropping does not remove entirely.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+Conv2DTranspose
+===============
+Base attributes
+---------------
+* result_t: NamedType
+
+  * The datatype (precision) of the output tensor.
+
+* weight_t: NamedType
+
+* bias_t: NamedType
+
+Type attributes
+---------------
+* index: int
+
+  * Internal node counter used for bookkeeping and variable/tensor naming.
+
+* in_height: int
+
+* in_width: int
+
+* out_height: int
+
+* out_width: int
+
+* n_chan: int
+
+* n_filt: int
+
+* filt_height: int
+
+* filt_width: int
+
+* stride_height: int
+
+* stride_width: int
+
+* dilation_height: int (Default: 1)
+
+* dilation_width: int (Default: 1)
+
+* crop_top: int
+
+* crop_left: int
+
+Weight attributes
+-----------------
+* weight: WeightVariable
+
+* bias: WeightVariable
+
+Configurable attributes
+-----------------------
+* trace: int (Default: False)
+
+  * Enables saving of layer output (tracing) when using hls_model.predict(...) or hls_model.trace(...)
+
+* result_t: NamedType
+
+  * The datatype (precision) of the output tensor.
+
+* weight_t: NamedType
+
+* bias_t: NamedType
+
+Backend-specific attributes
+---------------------------
+* accum_t: NamedType
+
+  * The datatype (precision) used to store intermediate results of the computation within the layer.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Quartus, Catapult, SymbolicExpression, Altera, Coyote, Libero, XLS
+
+* reuse_factor: int (Default: 1)
+
+  * The number of times each multiplier is used by controlling the amount of pipelining/unrolling. A lower number results in more parallelism and lower latency at the expense of the resources used. Reuse factor = 1 corresponds to all multiplications executed in parallel, and hence, the lowest possible latency.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Quartus, Catapult, SymbolicExpression, Altera, Coyote, Libero, XLS
+
+* parallelization_factor: int (Default: 1)
+
+  * The number of outputs computed in parallel. Essentially the number of multiplications of input window with the convolution kernel occurring in parallel. A higher number results in more parallelism (lower latency and II) at the expense of resources used. Currently only supported in io_parallel.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* slot_groups: tuple (Default: ())
+
+  * Set by the backend. The output is computed in blocks of stride positions, one block for each input position. For each kernel element, in the order of the rows of the weight matrix, the position in the block that it adds to.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* slot_shifts: tuple (Default: ())
+
+  * Set by the backend. For each kernel element, in the order of the rows of the weight matrix, how many blocks before the current block its input is, as (height, width).
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* n_buf: int (Default: 0)
+
+  * Set by the backend. The number of inputs multiplied for each block of outputs, which is the number of kernel elements times the number of input channels.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* first_block_height: int (Default: 0)
+
+  * Set by the backend. The index of the first block of outputs along this axis that cropping does not remove entirely.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* n_blocks_height: int (Default: 0)
+
+  * Set by the backend. The number of blocks of outputs along this axis that cropping does not remove entirely.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* first_block_width: int (Default: 0)
+
+  * Set by the backend. The index of the first block of outputs along this axis that cropping does not remove entirely.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
+* n_blocks_width: int (Default: 0)
+
+  * Set by the backend. The number of blocks of outputs along this axis that cropping does not remove entirely.
+
+  * Available in: Vivado, VivadoAccelerator, Vitis, VitisUnified, Coyote
+
 BatchNormalization
 ==================
 Base attributes

docs/ir/ir.rst

@@ -75,6 +75,48 @@ For the Vivado/Vitis backends, there is a dedicated ``io_parallel``/``Latency``
 The reuse factor (RF) is used to split the layer execution and reuse the existing module RF times. The RF also limits the number of multipliers in each module.
 The initiation interval scales as the RF. One limitation is that it assumes ``in_width`` is divisible by the RF.
 
+Transposed convolution
+**********************
+
+Transposed convolutions are the ``Conv1DTranspose`` and ``Conv2DTranspose`` layers in Keras, their quantized equivalents in QKeras-v3, and the
+``ConvTranspose1d`` and ``ConvTranspose2d`` layers in PyTorch. The Vivado and Vitis backends implement them for ``io_parallel`` and ``io_stream``
+with the ``Latency`` and ``Resource`` strategies. Strides, dilation, ``'valid'`` and ``'same'`` padding (any padding in PyTorch) and output padding
+are supported, as is an ``output_size`` passed to ``forward()`` in PyTorch when it is a constant. The data must be in the ``channels_last`` format,
+which PyTorch models are converted to as usual, and PyTorch layers with ``groups`` larger than 1 are not supported.
+
+With stride ``s``, each input pixel produces a block of ``s`` output pixels along each axis. Kernel element ``t`` adds to position
+``(t * dilation) % s`` of a block and reads the input ``(t * dilation) // s`` pixels before the block. The layer is computed one block at a time: the
+inputs of all kernel elements of a block, ``filt_height * filt_width * n_chan`` values, are multiplied by the weights, and each product is added to
+the output position of its kernel element. A block of up to ``s`` x ``s`` output pixels therefore costs as many multiplications as one output pixel of
+a convolution with the same kernel and stride 1, and no input is multiplied by an inserted zero. (``UpSampling`` followed by a convolution, which
+is sometimes used in place of a transposed convolution, is a different operation: it repeats the input pixels instead of inserting zeros.)
+
+io_parallel
+^^^^^^^^^^^
+
+As for the standard convolution, a generated function gathers the inputs of each block, and a second generated function writes the results of a
+block to the output and drops the positions that the padding removes. ``ReuseFactor`` is the number of clock cycles spent on one block; with the
+``Resource`` strategy it can be at most ``filt_height * filt_width * n_chan``. ``ParallelizationFactor`` is the number of blocks computed at the same
+time. The initiation interval is the number of blocks divided by ``ParallelizationFactor``, times ``ReuseFactor``, where the number of blocks is the
+number of blocks that hold output pixels, usually one per input pixel.
+
+The whole input and output are kept in registers, so this implementation suits small layers only. Vivado HLS 2020.1 stops at an output of more
+than 4096 values, the largest array it partitions. With Vitis HLS, C synthesis and co-simulation become slow for outputs of a few thousand values
+(14 and 77 minutes for an output of 16x16x8), and the ``Resource`` strategy uses many flip-flops (75 000 for the same layer). ``io_stream`` is the
+better choice for larger layers.
+
+io_stream
+^^^^^^^^^
+
+The layer reads and writes one pixel at a time, in two processes that run at the same time. The first process steps through the input pixels,
+keeps the earlier rows and columns that the kernel needs, and computes a block in ``ReuseFactor`` clock cycles at each step. The second writes the
+output pixels in row order, one per clock cycle, and keeps the blocks of a row of blocks until all of their rows are written. The initiation interval
+is the larger of the number of steps times ``ReuseFactor`` and the number of output pixels. The number of steps is the number of input pixels, or a
+few more when the output extends past the block of the last input pixel, for example with ``'valid'`` padding and a kernel larger than the stride.
+A ``ReuseFactor`` up to the number of output pixels divided by the number of steps, about ``stride_height * stride_width``, therefore lowers the
+number of multipliers without lengthening the interval. ``ParallelizationFactor`` has no effect. The depth of the FIFO between the two processes is
+the smallest that gives the shortest interval, found from a model of the clock cycles of the two processes.
+
 Recurrent Layers
 ----------------