In recent years, Neural Machine Translation (NMT) has achieved notable
results in various translation tasks. However, the word-by-word generation
manner determined by the autoregressive mechanism leads to high translation
latency of the NMT and restricts its low-latency applications.
Non-Autoregressive Neural Machine Translation (NAT) removes the autoregressive
mechanism and achieves significant decoding speedup through generating target
words independently and simultaneously. Nevertheless, NAT still takes the
word-level cross-entropy loss as the training objective, which is not optimal
because the output of NAT cannot be properly evaluated due to the multimodality
problem. In this article, we propose using sequence-level training objectives
to train NAT models, which evaluate the NAT outputs as a whole and correlates
well with the real translation quality. Firstly, we propose training NAT models
to optimize sequence-level evaluation metrics (e.g., BLEU) based on several
novel reinforcement algorithms customized for NAT, which outperforms the
conventional method by reducing the variance of gradient estimation. Secondly,
we introduce a novel training objective for NAT models, which aims to minimize
the Bag-of-Ngrams (BoN) difference between the model output and the reference
sentence. The BoN training objective is differentiable and can be calculated
efficiently without doing any approximations. Finally, we apply a three-stage
training strategy to combine these two methods to train the NAT model. We
validate our approach on four translation tasks (WMT14 En$\leftrightarrow$De,
WMT16 En$\leftrightarrow$Ro), which shows that our approach largely outperforms
NAT baselines and achieves remarkable performance on all translation tasks. The
source code is available at this https URL

本文提出了一种使用序列级训练目标来训练非自回归神经机器翻译模型的方法，通过几种为非自回归定制的新型强化算法，基于 BLEU 等序列级评价指标优化 NAT 模型的训练；介绍了一种基于 Bag-of-Ngrams（BoN）差异的 NAT 模型的新型训练目标；并通过三阶段训练策略将这两种方法结合起来应用，验证了该方法在多项任务中的鲜明表现。

非自回归神经机器翻译的序列级训练

Sequence-Level Training for Non-Autoregressive Neural Machine  Translation

Non-Autoregressive Neural Machine Translation (NAT) achieves significant
decoding speedup through generating target words independently and
simultaneously. However, in the context of non-autoregressive translation, the
word-level cross-entropy loss cannot model the target-side sequential
dependency properly, leading to its weak correlation with the translation
quality. As a result, NAT tends to generate influent translations with
over-translation and under-translation errors. In this paper, we propose to
train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model
output and the reference sentence. The bag-of-ngrams training objective is
differentiable and can be efficiently calculated, which encourages NAT to
capture the target-side sequential dependency and correlates well with the
translation quality. We validate our approach on three translation tasks and
show that our approach largely outperforms the NAT baseline by about 5.0 BLEU
scores on WMT14 En$\leftrightarrow$De and about 2.5 BLEU scores on WMT16
En$\leftrightarrow$Ro.

本文提出通过训练 NAT 最小化模型输出和参考句子之间的 N-Gram 差异来促进 NAT 捕捉目标方面的序列依赖性，并与翻译质量相关。在三个翻译任务上验证了我们的方法，结果显示我们的方法在 WMT14 En<->De 和 WMT16 En<->Ro 上比 NAT 基线大约 5.0 和 2.5 个 BLEU 分数。