In real-world systems, scaling has been critical for improving the
translation quality in autoregressive translation (AT), which however has not
been well studied for non-autoregressive translation (NAT). In this work, we
bridge the gap by systematically studying the impact of scaling on NAT
behaviors. Extensive experiments on six WMT benchmarks over two advanced NAT
models show that scaling can alleviate the commonly-cited weaknesses of NAT
models, resulting in better translation performance. To reduce the side-effect
of scaling on decoding speed, we empirically investigate the impact of NAT
encoder and decoder on the translation performance. Experimental results on the
large-scale WMT20 En-De show that the asymmetric architecture (e.g. bigger
encoder and smaller decoder) can achieve comparable performance with the
scaling model, while maintaining the superiority of decoding speed with
standard NAT models. To this end, we establish a new benchmark by validating
scaled NAT models on the scaled dataset, which can be regarded as a strong
baseline for future works. We release code, models and system outputs at
this https URL

本文系统研究了缩放对非自回归翻译的影响，发现缩放可以缓解其模型的弱点，并提升其翻译性能。同时，作者还研究了 NAT 编码器和解码器对翻译性能的影响，并建立了新的基准数据集作为未来研究的强有力基础线。

大规模重访非自回归翻译

Revisiting Non-Autoregressive Translation at Scale

Non-autoregressive (NAR) generation, which is first proposed in neural
machine translation (NMT) to speed up inference, has attracted much attention
in both machine learning and natural language processing communities. While NAR
generation can significantly accelerate inference speed for machine
translation, the speedup comes at the cost of sacrificed translation accuracy
compared to its counterpart, auto-regressive (AR) generation. In recent years,
many new models and algorithms have been designed/proposed to bridge the
accuracy gap between NAR generation and AR generation. In this paper, we
conduct a systematic survey with comparisons and discussions of various
non-autoregressive translation (NAT) models from different aspects.
Specifically, we categorize the efforts of NAT into several groups, including
data manipulation, modeling methods, training criterion, decoding algorithms,
and the benefit from pre-trained models. Furthermore, we briefly review other
applications of NAR models beyond machine translation, such as dialogue
generation, text summarization, grammar error correction, semantic parsing,
speech synthesis, and automatic speech recognition. In addition, we also
discuss potential directions for future exploration, including releasing the
dependency of KD, dynamic length prediction, pre-training for NAR, and wider
applications, etc. We hope this survey can help researchers capture the latest
progress in NAR generation, inspire the design of advanced NAR models and
algorithms, and enable industry practitioners to choose appropriate solutions
for their applications. The web page of this survey is at
https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

本文对不自回归生成（NAR）进行了系统的调查，并比较和讨论了不同方面的各种非自回归翻译（NAT）模型，包括数据处理、建模方法、训练标准、解码算法以及受到预训练模型的好处，同时简要回顾了 NAR 模型在机器翻译以外的其他应用，并讨论了未来探索的潜在方向。

神经机器翻译及其扩展中的非自回归生成调查

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Non-autoregressive machine translation (NAT) systems predict a sequence of
output tokens in parallel, achieving substantial improvements in generation
speed compared to autoregressive models. Existing NAT models usually rely on
the technique of knowledge distillation, which creates the training data from a
pretrained autoregressive model for better performance. Knowledge distillation
is empirically useful, leading to large gains in accuracy for NAT models, but
the reason for this success has, as of yet, been unclear. In this paper, we
first design systematic experiments to investigate why knowledge distillation
is crucial to NAT training. We find that knowledge distillation can reduce the
complexity of data sets and help NAT to model the variations in the output
data. Furthermore, a strong correlation is observed between the capacity of an
NAT model and the optimal complexity of the distilled data for the best
translation quality. Based on these findings, we further propose several
approaches that can alter the complexity of data sets to improve the
performance of NAT models. We achieve the state-of-the-art performance for the
NAT-based models, and close the gap with the autoregressive baseline on WMT14
En-De benchmark.

本文通过实验发现，知识蒸馏可以降低数据集的复杂度，帮助非自回归机器翻译模型 NART 更好地建模输出的变化，提高翻译质量，并提出多种方法来调整数据集复杂度以改进 NAT 模型的性能，达到了最先进的性能水平。