QuantizedLinear

class aimet_torch.nn.QuantizedLinear(*args, **kwargs)[source]

Quantized subclass of torch.nn.Linear

forward(self, input: torch.Tensor) → torch.Tensor[source]

Quantized forward of torch.nn.Linear.

The input(s), parameter(s) (if any), and output(s) will be quantized with self.input_quantizers, self.param_quantizers, and self.output_quantizers respectively.

For more information, see QuantizationMixin.

compute_encodings()

Compute encodings with additional protection against overflow/underflow.

Given

y = round((xW + b) * sx * sw / sy - zy)

HTP implements this equation as either eq 1 or eq 2, depending on hw version and other parameters
y = round(( xW + b ) * sx * sw / sy - zy)

= round(( xW + b ) * s’ - zy) … eq 1 = round(( xW + b - zy/s’ ) * s’ ) ≈ round(( xW + b - round(zy/s’) ) * s’ ) = round(( xW + b’ ) * s’ ) … eq 2

where:

name | description | dtype | equation |

|------|—————————|----------------------|—————————-| | x | input | uint8 or uint16 | round(x_float / sx + zx) | | W | weight | int4, int8, or int16 | round(W_float / sw) | | b | bias | int32 | round(b_float / (sx * sw)) | | y | output | dtype(x) | given above | | sx | input scale | float | - | | zx | input zero_point | dtype(x) | - | | sw | weight scale | float | - | | sy | output scale | float | - | | zy | output zero_point | dtype(y) | - | | s’ | requantization scale | float | sx * sw / sy | | b’ | combined accumulator bias | int32 | b - round(zy / s’) |

This function adjusts the weight scale to prevent 3 possible scenarios of overflow/underflow

  1. Bias Overflow - occurs if: |b| > 2**31 - bad because: Causes severe clipping error when exported as int32

  2. Requantization Scale Underflow - occurs if: s’ <= 2**-24 - bad because: If exponent e <= -24, HexNN misinterprets s’=2**e as 2**(e+32)

    due to internal type casting bug

  3. Accumulator Bias Overflow - occurs if: |b'| > 2**31 - bad because: HexNN internally stores b’ as int32

forward(*args, **kwargs)[source]

Computes a quantized version of the parent module’s forward method.

The forward() method should perform the following logic in order:

  1. Apply existing input quantizers to input tensors

  2. Apply existing param quantizers to the layer’s parameters

  3. Call the inherited torch.nn.Module forward method with quantized inputs and parameters

  4. Apply existing output quantizers to the outputs of the forward method

If all input, output, and parameter quantizers are None, this method will behave exactly the same as its parent module’s forward pass.