Knowledge distillation (KD) is a common approach to compress a teacher model
to reduce its inference cost and memory footprint, by training a smaller
student model. However, in the context of autoregressive language models (LMs),
we empirically find that larger teacher LMs might dramatically result in a
poorer student. In response to this problem, we conduct a series of analyses
and reveal that different tokens have different teaching modes, neglecting
which will lead to performance degradation. Motivated by this, we propose a
simple yet effective adaptive teaching approach (ATKD) to improve the KD. The
core of ATKD is to reduce rote learning and make teaching more diverse and
flexible. Extensive experiments on 8 LM tasks show that, with the help of ATKD,
various baseline KD methods can achieve consistent and significant performance
gains (up to +3.04% average score) across all model types and sizes. More
encouragingly, ATKD can improve the student model generalization effectively.

通过分析发现大型语言模型在教学学生模型时会导致性能下降，设计了一种自适应教学方法（ATKD）来改进知识蒸馏，并通过大量实验验证其在各种模型类型和规模上均能显著提高性能（平均得分增加至多 + 3.04%）。更重要的是，ATKD 能有效改善学生模型的泛化能力。