Knowledge Distillation (KD) is a predominant approach for BERT compression.
Previous KD-based methods focus on designing extra alignment losses for the
student model to mimic the behavior of the teacher model. These methods
transfer the knowledge in an indirect way. In this paper, we propose a novel
Weight-Inherited Distillation (WID), which directly transfers knowledge from
the teacher. WID does not require any additional alignment loss and trains a
compact student by inheriting the weights, showing a new perspective of
knowledge distillation. Specifically, we design the row compactors and column
compactors as mappings and then compress the weights via structural
re-parameterization. Experimental results on the GLUE and SQuAD benchmarks show
that WID outperforms previous state-of-the-art KD-based baselines. Further
analysis indicates that WID can also learn the attention patterns from the
teacher model without any alignment loss on attention distributions.

本文介绍了一种基于 Weight-Inherited Distillation (WID) 的知识蒸馏方法，通过直接从教师模型继承权重实现对学生模型的压缩，避免了传统方法中需要设计额外对齐损失的问题，实验结果表明 WID 在 GLUE 和 SQuAD 基准测试上的性能优于现有的知识蒸馏方法，并且可以在没有对齐损失的情况下学习到教师模型的注意力模式。