It is difficult for non-autoregressive translation (NAT) models to capture
the multi-modal distribution of target translations due to their conditional
independence assumption, which is known as the "multi-modality problem",
including the lexical multi-modality and the syntactic multi-modality. While
the first one has been well studied, the syntactic multi-modality brings severe
challenge to the standard cross entropy (XE) loss in NAT and is under studied.
In this paper, we conduct a systematic study on the syntactic multi-modality
problem. Specifically, we decompose it into short- and long-range syntactic
multi-modalities and evaluate several recent NAT algorithms with advanced loss
functions on both carefully designed synthesized datasets and real datasets. We
find that the Connectionist Temporal Classification (CTC) loss and the
Order-Agnostic Cross Entropy (OAXE) loss can better handle short- and
long-range syntactic multi-modalities respectively. Furthermore, we take the
best of both and design a new loss function to better handle the complicated
syntactic multi-modality in real-world datasets. To facilitate practical usage,
we provide a guide to use different loss functions for different kinds of
syntactic multi-modality.

该论文针对非自回归翻译模型中存在的语法多模式问题进行了系统研究。作者提出了连接时序分类（CTC）和无序交叉熵（OAXE）损失函数来处理不同范围的语法多模式，并设计了一种新的损失函数来更好地处理实际世界数据集中的复杂语法多模式。

非自回归机器翻译中的语法多模式研究

A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation

We propose a new training objective named order-agnostic cross entropy (OaXE)
for fully non-autoregressive translation (NAT) models. OaXE improves the
standard cross-entropy loss to ameliorate the effect of word reordering, which
is a common source of the critical multimodality problem in NAT. Concretely,
OaXE removes the penalty for word order errors, and computes the cross entropy
loss based on the best possible alignment between model predictions and target
tokens. Since the log loss is very sensitive to invalid references, we leverage
cross entropy initialization and loss truncation to ensure the model focuses on
a good part of the search space. Extensive experiments on major WMT benchmarks
show that OaXE substantially improves translation performance, setting new
state of the art for fully NAT models. Further analyses show that OaXE
alleviates the multimodality problem by reducing token repetitions and
increasing prediction confidence. Our code, data, and trained models are
available at this https URL

本文提出了一种新的训练目标，名为非自回归翻译（NAT）模型的无序交叉熵（OaXE），通过改善标准交叉熵损失来改善字词重排序的影响，并解决了 NAT 中的严重多模性问题，有效地提高了翻译性能，在 WMT 基准测试中取得了最新的最佳效果。