Non-autoregressive machine translation (NAT) systems predict a sequence of
output tokens in parallel, achieving substantial improvements in generation
speed compared to autoregressive models. Existing NAT models usually rely on
the technique of knowledge distillation, which creates the training data from a
pretrained autoregressive model for better performance. Knowledge distillation
is empirically useful, leading to large gains in accuracy for NAT models, but
the reason for this success has, as of yet, been unclear. In this paper, we
first design systematic experiments to investigate why knowledge distillation
is crucial to NAT training. We find that knowledge distillation can reduce the
complexity of data sets and help NAT to model the variations in the output
data. Furthermore, a strong correlation is observed between the capacity of an
NAT model and the optimal complexity of the distilled data for the best
translation quality. Based on these findings, we further propose several
approaches that can alter the complexity of data sets to improve the
performance of NAT models. We achieve the state-of-the-art performance for the
NAT-based models, and close the gap with the autoregressive baseline on WMT14
En-De benchmark.

本文通过实验发现，知识蒸馏可以降低数据集的复杂度，帮助非自回归机器翻译模型 NART 更好地建模输出的变化，提高翻译质量，并提出多种方法来调整数据集复杂度以改进 NAT 模型的性能，达到了最先进的性能水平。