Knowledge distillation, a technique for model compression and performance
enhancement, has gained significant traction in Neural Machine Translation
(NMT). However, existing research primarily focuses on empirical applications,
and there is a lack of comprehensive understanding of how student model
capacity, data complexity, and decoding strategies collectively influence
distillation effectiveness. Addressing this gap, our study conducts an in-depth
investigation into these factors, particularly focusing on their interplay in
word-level and sequence-level distillation within NMT. Through extensive
experimentation across datasets like IWSLT13 En$\rightarrow$Fr, IWSLT14
En$\rightarrow$De, and others, we empirically validate hypotheses related to
the impact of these factors on knowledge distillation. Our research not only
elucidates the significant influence of model capacity, data complexity, and
decoding strategies on distillation effectiveness but also introduces a novel,
optimized distillation approach. This approach, when applied to the IWSLT14
de$\rightarrow$en translation task, achieves state-of-the-art performance,
demonstrating its practical efficacy in advancing the field of NMT.

知识蒸馏是一种模型压缩和性能提升的技术，在神经机器翻译领域取得了显著进展。本研究通过深入调查学生模型容量、数据复杂性和解码策略在单词级和序列级蒸馏中的相互作用，验证了有关这些因素对知识蒸馏影响的假设，同时提出了一种新颖的优化蒸馏方法，实现了最先进的翻译性能，推动了神经机器翻译领域的发展。