State-of-the-art neural text generation models are typically trained to
maximize the likelihood of each token in the ground-truth sequence conditioned
on the previous target tokens. However, during inference, the model needs to
make a prediction conditioned on the tokens generated by itself. This
train-test discrepancy is referred to as exposure bias. Scheduled sampling is a
curriculum learning strategy that gradually exposes the model to its own
predictions during training to mitigate this bias. Most of the proposed
approaches design a scheduler based on training steps, which generally requires
careful tuning depending on the training setup. In this work, we introduce
Dynamic Scheduled Sampling with Imitation Loss (DySI), which maintains the
schedule based solely on the training time accuracy, while enhancing the
curriculum learning by introducing an imitation loss, which attempts to make
the behavior of the decoder indistinguishable from the behavior of a
teacher-forced decoder. DySI is universally applicable across training setups
with minimal tuning. Extensive experiments and analysis show that DySI not only
achieves notable improvements on standard machine translation benchmarks, but
also significantly improves the robustness of other text generation models.

本文介绍了一种基于 Dynamic Scheduled Sampling with Imitation Loss (DySI) 的神经文本生成模型，该模型通过引入模仿损失和动态调度表，解决了常见的曝光偏差问题，在标准机器翻译基准测试数据集上获得了显著的性能提升，并提高了其他文本生成模型的鲁棒性。