Deep quantization of neural networks (below eight bits) offers significant
promise in reducing their compute and storage cost. Albeit alluring, without
special techniques for training and optimization, deep quantization results in
significant accuracy loss. To further mitigate this loss, we propose a novel
sinusoidal regularization, called SinReQ1, for deep quantized training. SinReQ
adds a periodic term to the original objective function of the underlying
training algorithm. SinReQ exploits the periodicity, differentiability, and the
desired convexity profile in sinusoidal functions to automatically propel
weights towards values that are inherently closer to quantization levels.
Since, this technique does not require invasive changes to the training
procedure, SinReQ can harmoniously enhance quantized training algorithms.
SinReQ offers generality and flexibility as it is not limited to a certain
bitwidth or a uniform assignment of bitwidths across layers. We carry out
experimentation using the AlexNet, CIFAR-10, ResNet-18, ResNet-20, SVHN, and
VGG-11 DNNs with three to five bits for quantization and show the versatility
of SinReQ in enhancing multiple quantized training algorithms, DoReFa [32] and
WRPN [24]. Averaging across all the bit configurations shows that SinReQ closes
the accuracy gap between these two techniques and the full-precision runs by
32.4% and 27.5%, respectively. That is improving the absolute accuracy of
DoReFa and WRPN by 2.8% and 2.1%, respectively.

本文介绍了一种新的正弦正则化方法 - SinReQ1，它能有效地解决通过深度量化神经网络存在的精度损失问题，对多种深度量化训练算法的作用具有通用性和灵活性，并且能够改善 DoReFa 和 WRPN 的绝对精度。