We proposed Additive Powers-of-Two~(APoT) quantization, an efficient non-uniform quantization scheme that attends to the bell-shaped and long-tailed distribution of weights in neural networks. By constraining all quantization levels as a sum of several Powers-of-Two terms, APoT quantization enjoys overwhelming efficiency of computation and a good match with weights' distribution. A simple reparameterization on clipping function is applied to generate better-defined gradient for updating of optimal clipping threshold. Moreover, weight normalization is presented to refine the input distribution of weights to be more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models demonstrating the effectiveness of our proposed APoT quantization. For example, our 3-bit quantized ResNet-34 on ImageNet only drops 0.3% Top-1 and 0.2% Top-5 accuracy without bells and whistles, while the computation of our model is approximately 2x less than uniformly quantized neural networks.

本研究提出一种高效的非均匀量化方案，称为APoT quantization，该方案能够更好地匹配神经网络中权重和激活的分布，通过重新参数化剪裁函数来生成更好定义的梯度，并提供了一种细化权重分布的权重归一化方法，以使训练更加稳定和一致。实验结果表明，该方法胜过现有最先进方法，并且在ImageNet数据集上，经过4位量化的ResNet-50模型，准确率达到76.6％，同时与均匀量化模型相比，模型计算成本降低了22％。

Additive Powers-of-Two Quantization: 一种高效的非均匀化离散化神经网络方法