Non-autoregressive Transformers (NATs) reduce the inference latency of
Autoregressive Transformers (ATs) by predicting words all at once rather than
in sequential order. They have achieved remarkable progress in machine
translation as well as many other applications. However, a long-standing
challenge for NATs is the learning of multi-modality data distribution, which
is the main cause of the performance gap between NATs and ATs. In this paper,
we propose to ease the difficulty of modality learning via sampling from the
model distribution instead of the data distribution. We derive contrastive
constraints to stabilize the training process and integrate this resulting
objective with the state-of-the-art NAT architecture DA-Transformer. Our model
\method is examined on 3 different tasks, including machine translation, text
summarization, and paraphrasing with 5 benchmarks. Results show that our
approach outperforms previous non-autoregressive baselines by a significant
margin and establishes new state-of-the-art results for non-autoregressive
transformers on all the benchmarks.

本文提出了一种基于 DA-Transformer 模型和从模型分布中采点的对比约束的非自回归 Transformer 模型，通过这种方式减轻模态学习的难度，取得了在机器翻译，文本摘要和改写等多个基准数据集中显著的最新非自回归 Transformer 模型的性能。