Quantization reduces computation costs of neural networks but suffers from
performance degeneration. Is this accuracy drop due to the reduced capacity, or
inefficient training during the quantization procedure? After looking into the
gradient propagation process of neural networks by viewing the weights and
intermediate activations as random variables, we discover two critical rules
for efficient training. Recent quantization approaches violates the two rules
and results in degenerated convergence. To deal with this problem, we propose a
simple yet effective technique, named scale-adjusted training (SAT), to comply
with the discovered rules and facilitates efficient training. We also analyze
the quantization error introduced in calculating the gradient in the popular
parameterized clipping activation (PACT) technique. Through SAT together with
gradient-calibrated PACT, quantized models obtain comparable or even better
performance than their full-precision counterparts, achieving state-of-the-art
accuracy with consistent improvement over previous quantization methods on a
wide spectrum of models including MobileNet-V1/V2 and PreResNet-50.

提出了一种新的 quantization 方法，scale-adjusted training (SAT)，以更高效地训练神经网络，并分析了参数化剪辑激活（PACT）技术中梯度计算引入的量化误差。结合 SAT 和梯度校准的 PACT，量化模型可以达到与全精度模型相当甚至更好的性能，并在包括 MobileNet-V1 / V2 和 PreResNet-50 在内的大范围模型上持续改进，达到了最先进的精度。

神经网络量化的高效训练方法

Towards Efficient Training for Neural Network Quantization

Deep learning algorithms achieve high classification accuracy at the expense
of significant computation cost. To address this cost, a number of quantization
schemes have been proposed - but most of these techniques focused on quantizing
weights, which are relatively smaller in size compared to activations. This
paper proposes a novel quantization scheme for activations during training -
that enables neural networks to work well with ultra low precision weights and
activations without any significant accuracy degradation. This technique,
PArameterized Clipping acTivation (PACT), uses an activation clipping parameter
$\alpha$ that is optimized during training to find the right quantization
scale. PACT allows quantizing activations to arbitrary bit precisions, while
achieving much better accuracy relative to published state-of-the-art
quantization schemes. We show, for the first time, that both weights and
activations can be quantized to 4-bits of precision while still achieving
accuracy comparable to full precision networks across a range of popular models
and datasets. We also show that exploiting these reduced-precision
computational units in hardware can enable a super-linear improvement in
inferencing performance due to a significant reduction in the area of
accelerator compute engines coupled with the ability to retain the quantized
model and activation data in on-chip memories.

提出一种新的量化方案 PACT，可以在不降低准确性的情况下将激活量化至任意位精度，同时以 4 位精度量化权重和激活时能获得与完全精确网络相当的准确性，可以在硬件上获得超线性的推理性能提升。