Unsupervised cross-lingual pretraining has achieved strong results in neural
machine translation (NMT), by drastically reducing the need for large parallel
data. Most approaches adapt masked-language modeling (MLM) to
sequence-to-sequence architectures, by masking parts of the input and
reconstructing them in the decoder. In this work, we systematically compare
masking with alternative objectives that produce inputs resembling real (full)
sentences, by reordering and replacing words based on their context. We
pretrain models with different methods on English$\leftrightarrow$German,
English$\leftrightarrow$Nepali and English$\leftrightarrow$Sinhala monolingual
data, and evaluate them on NMT. In (semi-) supervised NMT, varying the
pretraining objective leads to surprisingly small differences in the finetuned
performance, whereas unsupervised NMT is much more sensitive to it. To
understand these results, we thoroughly study the pretrained models using a
series of probes and verify that they encode and use information in different
ways. We conclude that finetuning on parallel data is mostly sensitive to few
properties that are shared by most models, such as a strong decoder, in
contrast to unsupervised NMT that also requires models with strong
cross-lingual abilities.

本文研究无监督跨语言预训练在神经机器翻译中的应用，通过对编码器输入进行屏蔽或者基于上下文重新排序和替换等不同预训练方法的比较，发现无监督机器翻译对预训练目标的敏感度较高，与有监督机器翻译相比，更需要具有强大跨语言能力的编码器模型。