Quantization enables efficient acceleration of deep neural networks by
reducing model memory footprint and exploiting low-cost integer math hardware
units. Quantization maps floating-point weights and activations in a trained
model to low-bitwidth integer values using scale factors. Excessive
quantization, reducing precision too aggressively, results in accuracy
degradation. When scale factors are shared at a coarse granularity across many
dimensions of each tensor, effective precision of individual elements within
the tensor are limited. To reduce quantization-related accuracy loss, we
propose using a separate scale factor for each small vector of ($\approx$16-64)
elements within a single dimension of a tensor. To achieve an efficient
hardware implementation, the per-vector scale factors can be implemented with
low-bitwidth integers when calibrated using a two-level quantization scheme. We
find that per-vector scaling consistently achieves better inference accuracy at
low precision compared to conventional scaling techniques for popular neural
networks without requiring retraining. We also modify a deep learning
accelerator hardware design to study the area and energy overheads of
per-vector scaling support. Our evaluation demonstrates that per-vector scaled
quantization with 4-bit weights and activations achieves 37% area saving and
24% energy saving while maintaining over 75% accuracy for ResNet50 on ImageNet.
4-bit weights and 8-bit activations achieve near-full-precision accuracy for
both BERT-base and BERT-large on SQuAD while reducing area by 26% compared to
an 8-bit baseline.

本研究提出每个张量维度内小向量的单独比例因子缩放技术以降低量化相关的精度损失，有效提高了卷积神经网络的推理准确性并在深度学习加速器硬件设计上实现了硬件效率的提高和能源消耗的降低。