Beyond minimizing a single training loss, many deep learning estimation
pipelines rely on an auxiliary objective to quantify and encourage desirable
properties of the model (e.g. performance on another dataset, robustness,
agreement with a prior). Although the simplest approach to incorporating an
auxiliary loss is to sum it with the training loss as a regularizer, recent
works have shown that one can improve performance by blending the gradients
beyond a simple sum; this is known as gradient surgery. We cast the problem as
a constrained minimization problem where the auxiliary objective is minimized
among the set of minimizers of the training loss. To solve this bilevel
problem, we follow a parameter update direction that combines the training loss
gradient and the orthogonal projection of the auxiliary gradient to the
training gradient. In a setting where gradients come from mini-batches, we
explain how, using a moving average of the training loss gradients, we can
carefully maintain this critical orthogonality property. We demonstrate that
our method, Bloop, can lead to much better performances on NLP and vision
experiments than other gradient surgery methods without EMA.

通过混合梯度，我们提出了一种名称为 Bloop 的方法，该方法用于解决深度学习估计管道中的双层优化问题，并可在自然语言处理和视觉实验中取得比其他梯度手术方法更好的性能。

谨慎使用手术刀：用 EMA 改进梯度手术

Careful with that Scalpel: Improving Gradient Surgery with an EMA

For real-world deployment of automatic speech recognition (ASR), the system
is desired to be capable of fast inference while relieving the requirement of
computational resources. The recently proposed end-to-end ASR system based on
mask-predict with connectionist temporal classification (CTC), Mask-CTC,
fulfills this demand by generating tokens in a non-autoregressive fashion.
While Mask-CTC achieves remarkably fast inference speed, its recognition
performance falls behind that of conventional autoregressive (AR) systems. To
boost the performance of Mask-CTC, we first propose to enhance the encoder
network architecture by employing a recently proposed architecture called
Conformer. Next, we propose new training and decoding methods by introducing
auxiliary objective to predict the length of a partial target sequence, which
allows the model to delete or insert tokens during inference. Experimental
results on different ASR tasks show that the proposed approaches improve
Mask-CTC significantly, outperforming a standard CTC model (15.5% $\rightarrow$
9.1% WER on WSJ). Moreover, Mask-CTC now achieves competitive results to AR
models with no degradation of inference speed ($<$ 0.1 RTF using CPU). We also
show a potential application of Mask-CTC to end-to-end speech translation.

为了实现自动语音识别的实时应用，并降低计算资源需求，本文结合 Conformer 结构和辅助目标预测方法，提高了基于 Mask-CTC 的端到端自动语音识别系统识别准确性 17.5% 以上，同时使推理速度不下降，结果超越标准 CTC 模型。