Post-Training Quantization (PTQ) is a powerful technique for model
compression, reducing the precision of neural networks without additional
training overhead. Recent works have investigated adopting 8-bit floating-point
quantization (FP8) in the context of PTQ for model inference. However, the
exploration of floating-point formats smaller than 8 bits and their comparison
with integer quantization remains relatively limited. In this work, we present
minifloats, which are reduced-precision floating-point formats capable of
further reducing the memory footprint, latency, and energy cost of a model
while approaching full-precision model accuracy. Our work presents a novel PTQ
design-space exploration, comparing minifloat and integer quantization schemes
across a range of 3 to 8 bits for both weights and activations. We examine the
applicability of various PTQ techniques to minifloats, including weight
equalization, bias correction, SmoothQuant, gradient-based learned rounding,
and the GPTQ method. Our experiments validate the effectiveness of
low-precision minifloats when compared to their integer counterparts across a
spectrum of accuracy-precision trade-offs on a set of reference deep learning
vision workloads. Finally, we evaluate our results against an FPGA-based
hardware cost model, showing that integer quantization often remains the
Pareto-optimal option, given its relatively smaller hardware resource
footprint.

在这项研究中，我们通过对模型权重和激活函数使用 3 至 8 位的浮点数和整数量化方案，比较了浮点数量化和整数量化的适用性，验证了低精度 minifloats 在一系列精度 - 准确度权衡中相对于整数量化的有效性，并通过 FPGA 型号评估结果，表明整数量化通常是优选选择。