Despite strong performance in many sequence-to-sequence tasks, autoregressive
models trained with maximum likelihood estimation suffer from exposure bias,
i.e. the discrepancy between the ground-truth prefixes used during training and
the model-generated prefixes used at inference time. Scheduled sampling is a
simple and empirically successful approach which addresses this issue by
incorporating model-generated prefixes into training. However, it has been
argued that it is an inconsistent training objective leading to models ignoring
the prefixes altogether. In this paper, we conduct systematic experiments and
find that scheduled sampling, while it ameliorates exposure bias by increasing
model reliance on the input sequence, worsens performance when the prefix at
inference time is correct, a form of catastrophic forgetting. We propose to use
Elastic Weight Consolidation to better balance mitigating exposure bias with
retaining performance. Experiments on four IWSLT'14 and WMT'14 translation
datasets demonstrate that our approach alleviates catastrophic forgetting and
significantly outperforms maximum likelihood estimation and scheduled sampling
baselines.

通过系统实验，本文发现 MaxML 存在的曝光偏差问题是 “抽样计划” 的缺点，该计划加剧了当推理时间的前缀是正确的时的性能下降，即灾难性忘记。因此，提出使用 “弹性权重合并” 方法来更好地平衡减轻曝光偏差与保持性能。在四个翻译数据集上的实验表明，该方法缓解了灾难性忘记问题，并显著优于最大似然估计和计划抽样对照组。