We often use perturbations to regularize neural models. For neural
encoder-decoders, previous studies applied the scheduled sampling (Bengio et
al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations
but these methods require considerable computational time. Thus, this study
addresses the question of whether these approaches are efficient enough for
training time. We compare several perturbations in sequence-to-sequence
problems with respect to computational time. Experimental results show that the
simple techniques such as word dropout (Gal and Ghahramani, 2016) and random
replacement of input tokens achieve comparable (or better) scores to the
recently proposed perturbations, even though these simple methods are faster.
Our code is publicly available at
this https URL

研究比较序列到序列问题中常用的扰动方法，包括定期抽样、对抗性扰动、词随机替换等，结果表明速度更快的简单技术如词元删除、输入符号的随机替换等与新提出的扰动方法具有可比性或者更好的性能。

重新考虑编码器 - 解码器中的扰动以实现快速训练

Rethinking Perturbations in Encoder-Decoders for Fast Training

Many real world tasks exhibit rich structure that is repeated across
different parts of the state space or in time. In this work we study the
possibility of leveraging such repeated structure to speed up and regularize
learning. We start from the KL regularized expected reward objective which
introduces an additional component, a default policy. Instead of relying on a
fixed default policy, we learn it from data. But crucially, we restrict the
amount of information the default policy receives, forcing it to learn reusable
behaviors that help the policy learn faster. We formalize this strategy and
discuss connections to information bottleneck approaches and to the variational
EM algorithm. We present empirical results in both discrete and continuous
action domains and demonstrate that, for certain tasks, learning a default
policy alongside the policy can significantly speed up and improve learning.

研究如何通过限制收集到的信息，学习可重用行为作为默认策略，从而加速和规范学习过程，同时探讨了该策略与信息瓶颈方法和变分 EM 算法之间的联系，并在离散和连续动作领域提出了经验结果。