In this paper, we present a substantial step in better understanding the SOTA
sequence-to-sequence (Seq2Seq) pretraining for neural machine
translation~(NMT). We focus on studying the impact of the jointly pretrained
decoder, which is the main difference between Seq2Seq pretraining and previous
encoder-based pretraining approaches for NMT. By carefully designing
experiments on three language pairs, we find that Seq2Seq pretraining is a
double-edged sword: On one hand, it helps NMT models to produce more diverse
translations and reduce adequacy-related translation errors. On the other hand,
the discrepancies between Seq2Seq pretraining and NMT finetuning limit the
translation quality (i.e., domain discrepancy) and induce the over-estimation
issue (i.e., objective discrepancy). Based on these observations, we further
propose simple and effective strategies, named in-domain pretraining and input
adaptation to remedy the domain and objective discrepancies, respectively.
Experimental results on several language pairs show that our approach can
consistently improve both translation performance and model robustness upon
Seq2Seq pretraining.

本文主要研究了 Seq2Seq 预训练在神经机器翻译上的影响，发现 Seq2Seq 预训练可提高翻译多样性，但由于与 NMT 微调存在差异，限制了翻译质量和模型稳健性，进一步提出了域内预训练和输入适应策略可以缓解这些问题。实验结果显示，我们的方法可以一致地提高 Seq2Seq 预训练的翻译性能和模型稳健性。