Knowledge distillation~(KD) has been proved effective for compressing
large-scale pre-trained language models. However, existing methods conduct KD
statically, e.g., the student model aligns its output distribution to that of a
selected teacher model on the pre-defined training dataset. In this paper, we
explore whether a dynamic knowledge distillation that empowers the student to
adjust the learning procedure according to its competency, regarding the
student performance and learning efficiency. We explore the dynamical
adjustments on three aspects: teacher model adoption, data selection, and KD
objective adaptation. Experimental results show that (1) proper selection of
teacher model can boost the performance of student model; (2) conducting KD
with 10% informative instances achieves comparable performance while greatly
accelerates the training; (3) the student performance can be boosted by
adjusting the supervision contribution of different alignment objective. We
find dynamic knowledge distillation is promising and provide discussions on
potential future directions towards more efficient KD methods. Our code is
available at this https URL

本文研究知识蒸馏的动态方法是否能根据学生模型的能力，对三个方面进行调整，包括教师模型的采用，数据选择和蒸馏目标的调整，实验结果表明，动态知识蒸馏是有前途的，并提供了关于更有效 KD 方法的未来方向的讨论。