Knowledge Distillation (KD) has been validated as an effective model
compression technique for learning compact object detectors. Existing
state-of-the-art KD methods for object detection are mostly based on feature
imitation, which is generally observed to be better than prediction mimicking.
In this paper, we show that the inconsistency of the optimization objectives
between the ground-truth signals and distillation targets is the key reason for
the inefficiency of prediction mimicking. To alleviate this issue, we present a
simple yet effective distillation scheme, termed CrossKD, which delivers the
intermediate features of the student's detection head to the teacher's
detection head. The resulting cross-head predictions are then forced to mimic
the teacher's predictions. Such a distillation manner relieves the student's
head from receiving contradictory supervision signals from the ground-truth
annotations and the teacher's predictions, greatly improving the student's
detection performance. On MS COCO, with only prediction mimicking losses
applied, our CrossKD boosts the average precision of GFL ResNet-50 with 1x
training schedule from 40.2 to 43.7, outperforming all existing KD methods for
object detection. Code is available at this https URL

本文提出了一种名为 CrossKD 的知识蒸馏方法，该方法通过将学生的检测头的中间特征输出到老师的检测头，从而将跨头预测强制与老师的预测相同，以提高学生的检测性能，并在 MS COCO 上得到了验证，性能优于所有现有的目标检测方法。

交叉知识蒸馏：用于密集物体检测的交叉头知识蒸馏

CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection

In this paper, we investigate the knowledge distillation (KD) strategy for
object detection and propose an effective framework applicable to both
homogeneous and heterogeneous student-teacher pairs. The conventional feature
imitation paradigm introduces imitation masks to focus on informative
foreground areas while excluding the background noises. However, we find that
those methods fail to fully utilize the semantic information in all feature
pyramid levels, which leads to inefficiency for knowledge distillation between
FPN-based detectors. To this end, we propose a novel semantic-guided feature
imitation technique, which automatically performs soft matching between feature
pairs across all pyramid levels to provide the optimal guidance to the student.
To push the envelop even further, we introduce contrastive distillation to
effectively capture the information encoded in the relationship between
different feature regions. Finally, we propose a generalized detection KD
pipeline, which is capable of distilling both homogeneous and heterogeneous
detector pairs. Our method consistently outperforms the existing detection KD
techniques, and works when (1) components in the framework are used separately
and in conjunction; (2) for both homogeneous and heterogenous student-teacher
pairs and (3) on multiple detection benchmarks. With a powerful
X101-FasterRCNN-Instaboost detector as the teacher, R50-FasterRCNN reaches
44.0% AP, R50-RetinaNet reaches 43.3% AP and R50-FCOS reaches 43.1% AP on COCO
dataset.

研究如何应用知识蒸馏策略到目标检测中，同时提出了一种适用于同质和异质教师 - 学生的有效框架。通过提出一种新的语义引导特征模仿技术，以及引入对比蒸馏来有效地捕捉不同区域的信息编码关系，我们的方法在多个检测基准上持续优于现有的检测 KD 技术。

G-DetKD: 通过对比和语义引导特征模仿实现物体检测器的通用蒸馏框架

G-DetKD: Towards General Distillation Framework for Object Detectors via  Contrastive and Semantic-guided Feature Imitation

State-of-the-art CNN based recognition models are often computationally
prohibitive to deploy on low-end devices. A promising high level approach
tackling this limitation is knowledge distillation, which let small student
model mimic cumbersome teacher model's output to get improved generalization.
However, related methods mainly focus on simple task of classification while do
not consider complex tasks like object detection. We show applying the vanilla
knowledge distillation to detection model gets minor gain. To address the
challenge of distilling knowledge in detection model, we propose a fine-grained
feature imitation method exploiting the cross-location discrepancy of feature
response. Our intuition is that detectors care more about local near object
regions. Thus the discrepancy of feature response on the near object anchor
locations reveals important information of how teacher model tends to
generalize. We design a novel mechanism to estimate those locations and let
student model imitate the teacher on them to get enhanced performance. We first
validate the idea on a developed lightweight toy detector which carries
simplest notion of current state-of-the-art anchor based detection models on
challenging KITTI dataset, our method generates up to 15% boost of mAP for the
student model compared to the non-imitated counterpart. We then extensively
evaluate the method with Faster R-CNN model under various scenarios with common
object detection benchmark of Pascal VOC and COCO, imitation alleviates up to
74% performance drop of student model compared to teacher. Codes released at
this https URL

本文提出了一种基于细粒度特征模拟的知识蒸馏方法，以改善在复杂的目标检测任务中使用小型学生模型所遇到的性能瓶颈，通过估算目标附近的区域并让学生模型在这些区域上模仿教师模型，使得学生模型在性能表现上提高了 15% 到 74%。