Non-autoregressive Transformers (NATs) reduce the inference latency of
Autoregressive Transformers (ATs) by predicting words all at once rather than
in sequential order. They have achieved remarkable progress in machine
translation as well as many other applications. However, a long-standing
challenge for NATs is the learning of multi-modality data distribution, which
is the main cause of the performance gap between NATs and ATs. In this paper,
we propose to ease the difficulty of modality learning via sampling from the
model distribution instead of the data distribution. We derive contrastive
constraints to stabilize the training process and integrate this resulting
objective with the state-of-the-art NAT architecture DA-Transformer. Our model
\method is examined on 3 different tasks, including machine translation, text
summarization, and paraphrasing with 5 benchmarks. Results show that our
approach outperforms previous non-autoregressive baselines by a significant
margin and establishes new state-of-the-art results for non-autoregressive
transformers on all the benchmarks.

本文提出了一种基于 DA-Transformer 模型和从模型分布中采点的对比约束的非自回归 Transformer 模型，通过这种方式减轻模态学习的难度，取得了在机器翻译，文本摘要和改写等多个基准数据集中显著的最新非自回归 Transformer 模型的性能。

利用对比学习优化非自回归变换器

Optimizing Non-Autoregressive Transformers with Contrastive Learning

Non-autoregressive models achieve significant decoding speedup in neural
machine translation but lack the ability to capture sequential dependency.
Directed Acyclic Transformer (DA-Transformer) was recently proposed to model
sequential dependency with a directed acyclic graph. Consequently, it has to
apply a sequential decision process at inference time, which harms the global
translation accuracy. In this paper, we present a Viterbi decoding framework
for DA-Transformer, which guarantees to find the joint optimal solution for the
translation and decoding path under any length constraint. Experimental results
demonstrate that our approach consistently improves the performance of
DA-Transformer while maintaining a similar decoding speedup.

本文提出了一种基于维特比解码的 DA-Transformer 模型，能够在保持速度优势的同时提高翻译效果，适用于非自回归模型的神经机器翻译。

有向无环转换器的维特比解码在非自回归机器翻译中的应用

Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation

Transformer has achieved great success in the NLP field by composing various
advanced models like BERT and GPT. However, Transformer and its existing
variants may not be optimal in capturing token distances because the position
or distance embeddings used by these methods usually cannot keep the precise
information of real distances, which may not be beneficial for modeling the
orders and relations of contexts. In this paper, we propose DA-Transformer,
which is a distance-aware Transformer that can exploit the real distance. We
propose to incorporate the real distances between tokens to re-scale the raw
self-attention weights, which are computed by the relevance between attention
query and key. Concretely, in different self-attention heads the relative
distance between each pair of tokens is weighted by different learnable
parameters, which control the different preferences on long- or short-term
information of these heads. Since the raw weighted real distances may not be
optimal for adjusting self-attention weights, we propose a learnable sigmoid
function to map them into re-scaled coefficients that have proper ranges. We
first clip the raw self-attention weights via the ReLU function to keep
non-negativity and introduce sparsity, and then multiply them with the
re-scaled coefficients to encode real distance information into self-attention.
Extensive experiments on five benchmark datasets show that DA-Transformer can
effectively improve the performance of many tasks and outperform the vanilla
Transformer and its several variants.

本文提出了一种具备距离感知能力的 DA-Transformer 模型来捕捉输入 token 之间的真实距离信息，并将其用于自注意力计算中，实验证明该模型在五个基准数据集上的表现明显好于 Transformer 模型及其多个变体。