Knowledge Distillation (KD) emerges as one of the most promising compression
technologies to run advanced deep neural networks on resource-limited devices.
In order to train a small network (student) under the guidance of a large
network (teacher), the intuitive method is regularizing the feature maps or
logits of the student using the teacher's information. However, existing
methods either over-restrict the student to learn all information from the
teacher, which lead to some bad local minimum, or use various fancy and
elaborate modules to process and align features, which are complex and lack
generality. In this work, we proposed an abstract and general paradigm for the
KD task, referred to as DIMensionality Reduction KD (RdimKD), which solely
relies on dimensionality reduction, with a very minor modification to naive L2
loss. RdimKD straightforwardly utilizes a projection matrix to project both the
teacher's and student's feature maps onto a low-dimensional subspace, which are
then optimized during training. RdimKD achieves the goal in the simplest way
that not only does the student get valuable information from the teacher, but
it also ensures sufficient flexibility to adapt to the student's low-capacity
reality. Our extensive empirical findings indicate the effectiveness of RdimKD
across various learning tasks and diverse network architectures.

这篇研究提出了一种新的知识蒸馏范式，称为维度降低知识蒸馏（RdimKD），通过使用一个投影矩阵将大网络和小网络的特征图投影到低维子空间，并在训练过程中进行优化，既保证学生从老师获取有价值信息，又灵活适应学生的低容量实际情况。实证研究表明 RdimKD 在各种学习任务和不同网络架构中具有有效性。