Feature distillation is an effective way to improve the performance for a
smaller student model, which has fewer parameters and lower computation cost
compared to the larger teacher model. Unfortunately, there is a common obstacle
- the gap in semantic feature structure between the intermediate features of
teacher and student. The classic scheme prefers to transform intermediate
features by adding the adaptation module, such as naive convolutional,
attention-based or more complicated one. However, this introduces two problems:
a) The adaptation module brings more parameters into training. b) The
adaptation module with random initialization or special transformation isn't
friendly for distilling a pre-trained student. In this paper, we present
Matching Guided Distillation (MGD) as an efficient and parameter-free manner to
solve these problems. The key idea of MGD is to pose matching the teacher
channels with students' as an assignment problem. We compare three solutions of
the assignment problem to reduce channels from teacher features with partial
distillation loss. The overall training takes a coordinate-descent approach
between two optimization objects - assignments update and parameters update.
Since MGD only contains normalization or pooling operations with negligible
computation cost, it is flexible to plug into network with other distillation
methods.

本文提出了匹配引导蒸馏（Matching Guided Distillation，MGD）方法，该方法是一种高效的，不需要参数的方法，旨在解决适配模块带来的问题和随机初始化或特殊转换对预训练学生进行提炼的不利影响。MGD 的关键思想是将教师信道与学生信道的匹配作为分配问题，通过三种解决方案将教师信道的数量减少到学生信道，并使用部分蒸馏损失来更新分配。整个训练采用两个优化目标之间的协调下降方法。