Latent structure models are a powerful tool for modeling language data: they
can mitigate the error propagation and annotation bottleneck in pipeline
systems, while simultaneously uncovering linguistic insights about the data.
One challenge with end-to-end training of these models is the argmax operation,
which has null gradient. In this paper, we focus on surrogate gradients, a
popular strategy to deal with this problem. We explore latent structure
learning through the angle of pulling back the downstream learning objective.
In this paradigm, we discover a principled motivation for both the
straight-through estimator (STE) as well as the recently-proposed SPIGOT - a
variant of STE for structured models. Our perspective leads to new algorithms
in the same family. We empirically compare the known and the novel pulled-back
estimators against the popular alternatives, yielding new insight for
practitioners and revealing intriguing failure cases.

本文讨论了拉回下游学习目标方法来探索潜在结构学习的原理，从而发现了 STE 和 SPIGOT 的原则动机，这导致了相同家族中的新算法，并将已知的和新的拉回估计器与流行的选择进行了实证比较，为实践者提供了新的见识，并揭示了有趣的失败案例。

理解 SPIGOT 的机制：用作潜在结构学习的替代导数

Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent  Structure Learning

Training activation quantized neural networks involves minimizing a piecewise
constant function whose gradient vanishes almost everywhere, which is
undesirable for the standard back-propagation or chain rule. An empirical way
around this issue is to use a straight-through estimator (STE) (Bengio et al.,
2013) in the backward pass only, so that the "gradient" through the modified
chain rule becomes non-trivial. Since this unusual "gradient" is certainly not
the gradient of loss function, the following question arises: why searching in
its negative direction minimizes the training loss? In this paper, we provide
the theoretical justification of the concept of STE by answering this question.
We consider the problem of learning a two-linear-layer network with binarized
ReLU activation and Gaussian input data. We shall refer to the unusual
"gradient" given by the STE-modifed chain rule as coarse gradient. The choice
of STE is not unique. We prove that if the STE is properly chosen, the expected
coarse gradient correlates positively with the population gradient (not
available for the training), and its negation is a descent direction for
minimizing the population loss. We further show the associated coarse gradient
descent algorithm converges to a critical point of the population loss
minimization problem. Moreover, we show that a poor choice of STE leads to
instability of the training algorithm near certain local minima, which is
verified with CIFAR-10 experiments.

通过学习使用 STE 实现反向传播算法中的梯度下降，通过正确选择 STE 并验证其梯度与总体梯度正相关，进而解决了搜索负方向最小化训练损失的问题。研究还比较了不同 STE 算法对 CIFAR-10 数据集的训练结果和稳定性。

理解直通估计在训练激活量化神经网络中的作用

Understanding Straight-Through Estimator in Training Activation  Quantized Neural Nets

The Straight-Through Estimator (STE) is widely used for back-propagating
gradients through the quantization function, but the STE technique lacks a
complete theoretical understanding. We propose an alternative methodology
called alpha-blending (AB), which quantizes neural networks to low-precision
using stochastic gradient descent (SGD). Our method (AB) avoids STE
approximation by replacing the quantized weight in the loss function by an
affine combination of the quantized weight w_q and the corresponding
full-precision weight w with non-trainable scalar coefficient $\alpha$ and
$1-\alpha$. During training, $\alpha$ is gradually increased from 0 to 1; the
gradient updates to the weights are through the full-precision term,
$(1-\alpha)w$, of the affine combination; the model is converted from
full-precision to low-precision progressively. To evaluate the method, a 1-bit
BinaryNet on CIFAR10 dataset and 8-bits, 4-bits MobileNet v1, ResNet_50 v1/2 on
ImageNet dataset are trained using the alpha-blending approach, and the
evaluation indicates that AB improves top-1 accuracy by 0.9%, 0.82% and 2.93%
respectively compared to the results of STE based quantization.

该论文提出了一种 alpha-blending 替代 Straight-Through Estimator 的方法用于量化神经网络，该方法逐步将模型从全精度转化为低精度并在 CIFAR10 和 ImageNet 数据集上表现出更好的结果。