Low-bit deep neural networks (DNNs) become critical for embedded applications
due to their low storage requirement and computing efficiency. However, they
suffer much from the non-negligible accuracy drop. This paper proposes the
stochastic quantization (SQ) algorithm for learning accurate low-bit DNNs. The
motivation is due to the following observation. Existing training algorithms
approximate the real-valued elements/filters with low-bit representation all
together in each iteration. The quantization errors may be small for some
elements/filters, while are remarkable for others, which lead to inappropriate
gradient direction during training, and thus bring notable accuracy drop.
Instead, SQ quantizes a portion of elements/filters to low-bit with a
stochastic probability inversely proportional to the quantization error, while
keeping the other portion unchanged with full-precision. The quantized and
full-precision portions are updated with corresponding gradients separately in
each iteration. The SQ ratio is gradually increased until the whole network is
quantized. This procedure can greatly compensate the quantization error and
thus yield better accuracy for low-bit DNNs. Experiments show that SQ can
consistently and significantly improve the accuracy for different low-bit DNNs
on various datasets and various network structures.

本文介绍了一种用于学习准确的低比特深度神经网络的随机量化算法，其逐渐将网络量化，并可显著提高在不同数据集和结构上的低比特深度神经网络的准确度，适用于嵌入式应用。