It has been commonly observed that a teacher model with superior performance
does not necessarily result in a stronger student, highlighting a discrepancy
between current teacher training practices and effective knowledge transfer. In
order to enhance the guidance of the teacher training process, we introduce the
concept of distillation influence to determine the impact of distillation from
each training sample on the student's generalization ability. In this paper, we
propose Learning Good Teacher Matters (LGTM), an efficient training technique
for incorporating distillation influence into the teacher's learning process.
By prioritizing samples that are likely to enhance the student's generalization
ability, our LGTM outperforms 10 common knowledge distillation baselines on 6
text classification tasks in the GLUE benchmark.

本文提出了一种名为 LGTM 的有效教师培训技术，该技术通过引入蒸馏影响概念来确定每个训练样本对学生泛化能力的影响，通过优先选择可能增强学生泛化能力的样本，在 GLUE 基准测试数据集的 6 项文本分类任务中，LGTM 比其他 10 种常见的知识蒸馏基线表现更好。