Knowledge distillation (KD) is essential for training non-autoregressive
translation (NAT) models by reducing the complexity of the raw data with an
autoregressive teacher model. In this study, we empirically show that as a side
effect of this training, the lexical choice errors on low-frequency words are
propagated to the NAT model from the teacher model. To alleviate this problem,
we propose to expose the raw data to NAT models to restore the useful
information of low-frequency words, which are missed in the distilled data. To
this end, we introduce an extra Kullback-Leibler divergence term derived by
comparing the lexical choice of NAT model and that embedded in the raw data.
Experimental results across language pairs and model architectures demonstrate
the effectiveness and universality of the proposed approach. Extensive analyses
confirm our claim that our approach improves performance by reducing the
lexical choice errors on low-frequency words. Encouragingly, our approach
pushes the SOTA NAT performance on the WMT14 English-German and WMT16
Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. The
source code will be released.

本文通过知识蒸馏的方式训练非自回归翻译模型，并指出在低频词方面，自回归模型的词汇选择误差会传递给非自回归模型，同时提出了一个额外的 KL 散度项以减少低频词方面的词汇选择误差并通过实验验证了其有效性和普适性。