This work proposes a novel Deep Neural Network (DNN) quantization framework,
namely RMSMP, with a Row-wise Mixed-Scheme and Multi-Precision approach.
Specifically, this is the first effort to assign mixed quantization schemes and
multiple precisions within layers -- among rows of the DNN weight matrix, for
simplified operations in hardware inference, while preserving accuracy.
Furthermore, this paper makes a different observation from the prior work that
the quantization error does not necessarily exhibit the layer-wise sensitivity,
and actually can be mitigated as long as a certain portion of the weights in
every layer are in higher precisions. This observation enables layer-wise
uniformality in the hardware implementation towards guaranteed inference
acceleration, while still enjoying row-wise flexibility of mixed schemes and
multiple precisions to boost accuracy. The candidates of schemes and precisions
are derived practically and effectively with a highly hardware-informative
strategy to reduce the problem search space. With the offline determined ratio
of different quantization schemes and precisions for all the layers, the RMSMP
quantization algorithm uses the Hessian and variance-based method to
effectively assign schemes and precisions for each row. The proposed RMSMP is
tested for the image classification and natural language processing (BERT)
applications and achieves the best accuracy performance among state-of-the-arts
under the same equivalent precisions. The RMSMP is implemented on FPGA devices,
achieving 3.65x speedup in the end-to-end inference time for ResNet-18 on
ImageNet, compared with the 4-bit Fixed-point baseline.

本文提出了一种新颖的深度神经网络量化框架 ——RMSMP，采用行混合方案和多精度方法，在硬件推理中简化操作，同时保持准确性。通过对所有层进行线下确定量化方案，RMSMP 量化算法使用 Hessian 和基于方差的方法为每行分配精度和方案，并在 FPGA 设备上实现，与 4 位定点基线相比，在 ImageNet 上对 ResNet-18 的端到端推理时间实现了 3.65 倍加速。