Knowledge distillation, a technique for model compression and performance enhancement, has gained significant traction in Neural Machine Translation (NMT). However, existing research primarily focuses on empirical applications, and there is a lack of comprehensive understanding of how student model capacity, data complexity, and decoding strategies collectively influence distillation effectiveness. Addressing this gap, our study conducts an in-depth investigation into these factors, particularly focusing on their interplay in word-level and sequence-level distillation within NMT. Through extensive experimentation across datasets like IWSLT13 En$\rightarrow$Fr, IWSLT14 En$\rightarrow$De, and others, we empirically validate hypotheses related to the impact of these factors on knowledge distillation. Our research not only elucidates the significant influence of model capacity, data complexity, and decoding strategies on distillation effectiveness but also introduces a novel, optimized distillation approach. This approach, when applied to the IWSLT14 de$\rightarrow$en translation task, achieves state-of-the-art performance, demonstrating its practical efficacy in advancing the field of NMT.

知识蒸馏是一种模型压缩和性能提升的技术，在神经机器翻译领域取得了显著进展。本研究通过深入调查学生模型容量、数据复杂性和解码策略在单词级和序列级蒸馏中的相互作用，验证了有关这些因素对知识蒸馏影响的假设，同时提出了一种新颖的优化蒸馏方法，实现了最先进的翻译性能，推动了神经机器翻译领域的发展。

知识蒸馏的关键因素剖析