We consider learning deep neural networks (DNNs) that consist of
low-precision weights and activations for efficient inference of fixed-point
operations. In training low-precision networks, gradient descent in the
backward pass is performed with high-precision weights while quantized
low-precision weights and activations are used in the forward pass to calculate
the loss function for training. Thus, the gradient descent becomes suboptimal,
and accuracy loss follows. In order to reduce the mismatch in the forward and
backward passes, we utilize mean squared quantization error (MSQE)
regularization. In particular, we propose using a learnable regularization
coefficient with the MSQE regularizer to reinforce the convergence of
high-precision weights to their quantized values. We also investigate how
partial L2 regularization can be employed for weight pruning in a similar
manner. Finally, combining weight pruning, quantization, and entropy coding, we
establish a low-precision DNN compression pipeline. In our experiments, the
proposed method yields low-precision MobileNet and ShuffleNet models on
ImageNet classification with the state-of-the-art compression ratios of 7.13
and 6.79, respectively. Moreover, we examine our method for image super
resolution networks to produce 8-bit low-precision models at negligible
performance loss.

本文提出了一种使用低精度权重和操作的 DNN 学习方法，利用可学习的正则化系数来加强高精度权重收敛到量化值的能力，并研究了如何通过权重剪枝、量化和熵编码来建立低精度 DNN 压缩管道，实验结果显示该方法可以在 ImageNet 分类和图像超分辨率网络的任务中取得与准确性相对可接受的最新压缩比。