Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowledge distillation methods suffer from two limitations. First, the student model simply imitates the teacher's behavior while ignoring the underlying reasoning. Second, these methods usually focus on the transfer of sophisticated model-specific knowledge but overlook data-specific knowledge. In this paper, we present a novel attribution-driven knowledge distillation approach, which explores the token-level rationale behind the teacher model based on Integrated Gradients (IG) and transfers attribution knowledge to the student model. To enhance the knowledge transfer of model reasoning and generalization, we further explore multi-view attribution distillation on all potential decisions of the teacher. Comprehensive experiments are conducted with BERT on the GLUE benchmark. The experimental results demonstrate the superior performance of our approach to several state-of-the-art methods.

本文提出了一种基于Integrated Gradients的新颖的归因驱动知识蒸馏方法，它探索了教师模型背后的token-level解释，并将知识转移给学生模型，进一步探索了多视角归因蒸馏。实验证明我们的方法在GLUE基准测试中具有比其他现有方法更好的性能。

AD-KD: 基于属性的知识蒸馏用于语言模型压缩