Fully quantized training (FQT), which uses low-bitwidth hardware by
quantizing the activations, weights, and gradients of a neural network model,
is a promising approach to accelerate the training of deep neural networks. One
major challenge with FQT is the lack of theoretical understanding, in
particular of how gradient quantization impacts convergence properties. In this
paper, we address this problem by presenting a statistical framework for
analyzing FQT algorithms. We view the quantized gradient of FQT as a stochastic
estimator of its full precision counterpart, a procedure known as
quantization-aware training (QAT). We show that the FQT gradient is an unbiased
estimator of the QAT gradient, and we discuss the impact of gradient
quantization on its variance. Inspired by these theoretical results, we develop
two novel gradient quantizers, and we show that these have smaller variance
than the existing per-tensor quantizer. For training ResNet-50 on ImageNet, our
5-bit block Householder quantizer achieves only 0.5% validation accuracy loss
relative to QAT, comparable to the existing INT8 baseline.

本论文提出了一个用于分析全量化训练算法的统计框架，并探讨了梯度量化对其收敛性的影响。作者开发了两个新的梯度量化器，并展示了这些量化器相对于现有的每个张量量化器具有更小的方差。

一种深度神经网络低比特宽度训练的统计框架

A Statistical Framework for Low-bitwidth Training of Deep Neural  Networks

Motivated by Danskin's theorem, gradient-based methods have been applied with
empirical success to solve minimax problems that involve non-convex outer
minimization and non-concave inner maximization. On the other hand, recent work
has demonstrated that Evolution Strategies (ES) algorithms are stochastic
gradient approximators that seek robust solutions. In this paper, we address
black-box (gradient-free) minimax problems that have long been tackled in a
coevolutionary setup. To this end and guaranteed by Danskin's theorem, we
employ ES as a stochastic estimator for the descent direction. The proposed
approach is validated on a collection of black-box minimax problems. Based on
our experiments, our method's performance is comparable with its coevolutionary
counterparts and favorable for high-dimensional problems. Its efficacy is
demonstrated on a real-world application.

本文提出了一种使用进化策略作为梯度估计器的黑盒极小极大问题的求解方法，并在实验中展示了该方法的有效性。