To make deep neural networks feasible in resource-constrained environments
(such as mobile devices), it is beneficial to quantize models by using
low-precision weights. One common technique for quantizing neural networks is
the straight-through gradient method, which enables back-propagation through
the quantization mapping. Despite its empirical success, little is understood
about why the straight-through gradient method works.
Building upon a novel observation that the straight-through gradient method
is in fact identical to the well-known Nesterov's dual-averaging algorithm on a
quantization constrained optimization problem, we propose a more principled
alternative approach, called ProxQuant, that formulates quantized network
training as a regularized learning problem instead and optimizes it via the
prox-gradient method. ProxQuant does back-propagation on the underlying
full-precision vector and applies an efficient prox-operator in between
stochastic gradient steps to encourage quantizedness. For quantizing ResNets
and LSTMs, ProxQuant outperforms state-of-the-art results on binary
quantization and is on par with state-of-the-art on multi-bit quantization. For
binary quantization, our analysis shows both theoretically and experimentally
that ProxQuant is more stable than the straight-through gradient method (i.e.
BinaryConnect), challenging the indispensability of the straight-through
gradient method and providing a powerful alternative.

通过发现直通渐变法与 Nesterov 双平均算法相同，提出了一种更加合理的替代方法 ProxQuant，将基于量化网络的训练问题转化为规则化学习问题，并通过 Prox - 梯度方法进行优化。对于二元量化，ProxQuant 在稳定性上优于直通渐变法，并提供了一种强有力的替代方法。在将 ResNets 和 LSTMs 量化为二元和多位之后，ProxQuant 能在二元量化上实现优于现有水平的结果，并且在多位量化上与现有技术持平。