Pre-trained sequence-to-sequence models have significantly improved Neural
Machine Translation (NMT). Different from prior works where pre-trained models
usually adopt an unidirectional decoder, this paper demonstrates that
pre-training a sequence-to-sequence model but with a bidirectional decoder can
produce notable performance gains for both Autoregressive and
Non-autoregressive NMT. Specifically, we propose CeMAT, a conditional masked
language model pre-trained on large-scale bilingual and monolingual corpora in
many languages. We also introduce two simple but effective methods to enhance
the CeMAT, aligned code-switching & masking and dynamic dual-masking. We
conduct extensive experiments and show that our CeMAT can achieve significant
performance improvement for all scenarios from low- to extremely high-resource
languages, i.e., up to +14.4 BLEU on low resource and +7.9 BLEU improvements on
average for Autoregressive NMT. For Non-autoregressive NMT, we demonstrate it
can also produce consistent performance gains, i.e., up to +5.3 BLEU. To the
best of our knowledge, this is the first work to pre-train a unified model for
fine-tuning on both NMT tasks. Code, data, and pre-trained models are available
at this https URL.

本文介绍了一种 CeMAT 方法，即一种在双语和单语语料库上进行预训练的条件掩码语言模型，其中包括双向解码器，通过该方法可以在低资源及高资源语言下分别提高 14.4 BLEU 和 7.9 BLEU 及能显著提高 Autoregressive NMT 的性能。此外，本文还首次在 NMT 任务上使用了统一预训练模型，并提供相关数据和模型。