Knowledge distillation, transferring knowledge from a teacher model to a
student model, has emerged as a powerful technique in neural machine
translation for compressing models or simplifying training targets. Knowledge
distillation encompasses two primary methods: sentence-level distillation and
token-level distillation. In sentence-level distillation, the student model is
trained to align with the output of the teacher model, which can alleviate the
training difficulty and give student model a comprehensive understanding of
global structure. Differently, token-level distillation requires the student
model to learn the output distribution of the teacher model, facilitating a
more fine-grained transfer of knowledge. Studies have revealed divergent
performances between sentence-level and token-level distillation across
different scenarios, leading to the confusion on the empirical selection of
knowledge distillation methods. In this study, we argue that token-level
distillation, with its more complex objective (i.e., distribution), is better
suited for ``simple'' scenarios, while sentence-level distillation excels in
``complex'' scenarios. To substantiate our hypothesis, we systematically
analyze the performance of distillation methods by varying the model size of
student models, the complexity of text, and the difficulty of decoding
procedure. While our experimental results validate our hypothesis, defining the
complexity level of a given scenario remains a challenging task. So we further
introduce a novel hybrid method that combines token-level and sentence-level
distillation through a gating mechanism, aiming to leverage the advantages of
both individual methods. Experiments demonstrate that the hybrid method
surpasses the performance of token-level or sentence-level distillation methods
and the previous works by a margin, demonstrating the effectiveness of the
proposed hybrid method.

通过改进的混合方法，将知识蒸馏技术应用于神经机器翻译，以在不同的情境中提高模型性能和压缩模型。