Knowledge distillation (KD) is a promising technique for model compression in
neural machine translation. However, where the knowledge hides in KD is still
not clear, which may hinder the development of KD. In this work, we first
unravel this mystery from an empirical perspective and show that the knowledge
comes from the top-1 predictions of teachers, which also helps us build a
potential connection between word- and sequence-level KD. Further, we point out
two inherent issues in vanilla word-level KD based on this finding. Firstly,
the current objective of KD spreads its focus to whole distributions to learn
the knowledge, yet lacks special treatment on the most crucial top-1
information. Secondly, the knowledge is largely covered by the golden
information due to the fact that most top-1 predictions of teachers overlap
with ground-truth tokens, which further restricts the potential of KD. To
address these issues, we propose a novel method named \textbf{T}op-1
\textbf{I}nformation \textbf{E}nhanced \textbf{K}nowledge \textbf{D}istillation
(TIE-KD). Specifically, we design a hierarchical ranking loss to enforce the
learning of the top-1 information from the teacher. Additionally, we develop an
iterative KD procedure to infuse more additional knowledge by distilling on the
data without ground-truth targets. Experiments on WMT'14 English-German, WMT'14
English-French and WMT'16 English-Romanian demonstrate that our method can
respectively boost Transformer$_{base}$ students by +1.04, +0.60 and +1.11 BLEU
scores and significantly outperform the vanilla word-level KD baseline.
Besides, our method shows higher generalizability on different teacher-student
capacity gaps than existing KD techniques.

本文研究神经机器翻译中知识蒸馏的技术，发现知识来源于教师的 top-1 预测，进一步提出一种名为 TIE-KD 的方法用于增强知识蒸馏，包含了层次排序损失和迭代蒸馏等措施，实验证明 TIE-KD 优于基准模型，具有更高的潜力和泛化性能。