Pre-trained sequence-to-sequence models have significantly improved Neural
Machine Translation (NMT). Different from prior works where pre-trained models
usually adopt an unidirectional decoder, this paper demonstrates that
pre-training a sequence-to-sequence model but with a bidirectional decoder can
produce notable performance gains for both Autoregressive and
Non-autoregressive NMT. Specifically, we propose CeMAT, a conditional masked
language model pre-trained on large-scale bilingual and monolingual corpora in
many languages. We also introduce two simple but effective methods to enhance
the CeMAT, aligned code-switching & masking and dynamic dual-masking. We
conduct extensive experiments and show that our CeMAT can achieve significant
performance improvement for all scenarios from low- to extremely high-resource
languages, i.e., up to +14.4 BLEU on low resource and +7.9 BLEU improvements on
average for Autoregressive NMT. For Non-autoregressive NMT, we demonstrate it
can also produce consistent performance gains, i.e., up to +5.3 BLEU. To the
best of our knowledge, this is the first work to pre-train a unified model for
fine-tuning on both NMT tasks. Code, data, and pre-trained models are available
at this https URL.

本文介绍了一种 CeMAT 方法，即一种在双语和单语语料库上进行预训练的条件掩码语言模型，其中包括双向解码器，通过该方法可以在低资源及高资源语言下分别提高 14.4 BLEU 和 7.9 BLEU 及能显著提高 Autoregressive NMT 的性能。此外，本文还首次在 NMT 任务上使用了统一预训练模型，并提供相关数据和模型。

神经机器翻译的通用条件掩码语言预训练

Universal Conditional Masked Language Pre-training for Neural Machine Translation

Neural Machine Translation(NMT) models are usually trained via unidirectional
decoder which corresponds to optimizing one-step-ahead prediction. However,
this kind of unidirectional decoding framework may incline to focus on local
structure rather than global coherence. To alleviate this problem, we propose a
novel method, Self-Knowledge Distillation with Bidirectional Decoder for Neural
Machine Translation(SBD-NMT). We deploy a backward decoder which can act as an
effective regularization method to the forward decoder. By leveraging the
backward decoder's information about the longer-term future, distilling
knowledge learned in the backward decoder can encourage auto-regressive NMT
models to plan ahead. Experiments show that our method is significantly better
than the strong Transformer baselines on multiple machine translation data
sets.

介绍了一种使用自我知识蒸馏和双向解码的神经机器翻译模型，使用这种方法可以鼓励自回归 NMT 模型提前规划，实验表明该方法在多个机器翻译数据集上比强 Transformer 基线方法显著优秀。

自我知识蒸馏中的前后向查看：双向解码器用于神经机器翻译

Look Backward and Forward: Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translation

We consider the problem of collectively detecting multiple events,
particularly in cross-sentence settings. The key to dealing with the problem is
to encode semantic information and model event inter-dependency at a
document-level. In this paper, we reformulate it as a Seq2Seq task and propose
a Multi-Layer Bidirectional Network (MLBiNet) to capture the document-level
association of events and semantic information simultaneously. Specifically, a
bidirectional decoder is firstly devised to model event inter-dependency within
a sentence when decoding the event tag vector sequence. Secondly, an
information aggregation module is employed to aggregate sentence-level semantic
and event tag information. Finally, we stack multiple bidirectional decoders
and feed cross-sentence information, forming a multi-layer bidirectional
tagging architecture to iteratively propagate information across sentences. We
show that our approach provides significant improvement in performance compared
to the current state-of-the-art results.

本文提出一种基于 Seq2Seq 的多层双向网络（MLBiNet）模型，用于在跨句子的情况下集体检测多个事件，并利用信息聚合模块和多层双向编码器来编码语义信息和事件之间的关联，从而提高跨句子检测的性能。