Knowledge distillation transfers knowledge from the teacher network to the
student one, with the goal of greatly improving the performance of the student
network. Previous methods mostly focus on proposing feature transformation and
loss functions between the same level's features to improve the effectiveness.
We differently study the factor of connection path cross levels between teacher
and student networks, and reveal its great importance. For the first time in
knowledge distillation, cross-stage connection paths are proposed. Our new
review mechanism is effective and structurally simple. Our finally designed
nested and compact framework requires negligible computation overhead, and
outperforms other methods on a variety of tasks. We apply our method to
classification, object detection, and instance segmentation tasks. All of them
witness significant student network performance improvement. Code is available
at this https URL

本研究在知识蒸馏中不同于先前方法，提出了跨层连接路径的重要性，通过新的重审机制设计了嵌套紧凑的框架，极大地提高了学生网络的性能，在分类、目标检测和实例分割等任务中均获得了显著的成果。