Quantization-aware training comes with a fundamental challenge: the
derivative of quantization functions such as rounding are zero almost
everywhere and nonexistent elsewhere. Various differentiable approximations of
quantization functions have been proposed to address this issue. In this paper,
we prove that when the learning rate is sufficiently small, a large class of
weight gradient estimators is equivalent with the straight through estimator
(STE). Specifically, after swapping in the STE and adjusting both the weight
initialization and the learning rate in SGD, the model will train in almost
exactly the same way as it did with the original gradient estimator. Moreover,
we show that for adaptive learning rate algorithms like Adam, the same result
can be seen without any modifications to the weight initialization and learning
rate. We experimentally show that these results hold for both a small
convolutional model trained on the MNIST dataset and for a ResNet50 model
trained on ImageNet.

当学习速率足够小的时候，我们通过实验证明了一类大规模的权重梯度估计器等价于直通估计器 (STE)，无需改变权重初始化和学习速率，可用于处理量化感知训练中的梯度问题。