Knowledge Distillation (KD) has been validated as an effective model
compression technique for learning compact object detectors. Existing
state-of-the-art KD methods for object detection are mostly based on feature
imitation, which is generally observed to be better than prediction mimicking.
In this paper, we show that the inconsistency of the optimization objectives
between the ground-truth signals and distillation targets is the key reason for
the inefficiency of prediction mimicking. To alleviate this issue, we present a
simple yet effective distillation scheme, termed CrossKD, which delivers the
intermediate features of the student's detection head to the teacher's
detection head. The resulting cross-head predictions are then forced to mimic
the teacher's predictions. Such a distillation manner relieves the student's
head from receiving contradictory supervision signals from the ground-truth
annotations and the teacher's predictions, greatly improving the student's
detection performance. On MS COCO, with only prediction mimicking losses
applied, our CrossKD boosts the average precision of GFL ResNet-50 with 1x
training schedule from 40.2 to 43.7, outperforming all existing KD methods for
object detection. Code is available at this https URL

本文提出了一种名为 CrossKD 的知识蒸馏方法，该方法通过将学生的检测头的中间特征输出到老师的检测头，从而将跨头预测强制与老师的预测相同，以提高学生的检测性能，并在 MS COCO 上得到了验证，性能优于所有现有的目标检测方法。