Knowledge distillation (KD) has shown potential for learning compact models
in dense object detection. However, the commonly used softmax-based
distillation ignores the absolute classification scores for individual
categories. Thus, the optimum of the distillation loss does not necessarily
lead to the optimal student classification scores for dense object detectors.
This cross-task protocol inconsistency is critical, especially for dense object
detectors, since the foreground categories are extremely imbalanced. To address
the issue of protocol differences between distillation and classification, we
propose a novel distillation method with cross-task consistent protocols,
tailored for the dense object detection. For classification distillation, we
address the cross-task protocol inconsistency problem by formulating the
classification logit maps in both teacher and student models as multiple
binary-classification maps and applying a binary-classification distillation
loss to each map. For localization distillation, we design an IoU-based
Localization Distillation Loss that is free from specific network structures
and can be compared with existing localization distillation losses. Our
proposed method is simple but effective, and experimental results demonstrate
its superiority over existing methods. Code is available at
this https URL

我们提出了一种用于密集物体检测的新的知识蒸馏方法，解决了蒸馏和分类之间协议差异的问题，并证明其优于现有方法。

稠密物体检测中蒸馏的跨任务协议一致性桥接

Bridging Cross-task Protocol Inconsistency for Distillation in Dense  Object Detection

Previous knowledge distillation (KD) methods for object detection mostly
focus on feature imitation instead of mimicking the prediction logits due to
its inefficiency in distilling the localization information. In this paper, we
investigate whether logit mimicking always lags behind feature imitation.
Towards this goal, we first present a novel localization distillation (LD)
method which can efficiently transfer the localization knowledge from the
teacher to the student. Second, we introduce the concept of valuable
localization region that can aid to selectively distill the classification and
localization knowledge for a certain region. Combining these two new
components, for the first time, we show that logit mimicking can outperform
feature imitation and the absence of localization distillation is a critical
reason for why logit mimicking underperforms for years. The thorough studies
exhibit the great potential of logit mimicking that can significantly alleviate
the localization ambiguity, learn robust feature representation, and ease the
training difficulty in the early stage. We also provide the theoretical
connection between the proposed LD and the classification KD, that they share
the equivalent optimization effect. Our distillation scheme is simple as well
as effective and can be easily applied to both dense horizontal object
detectors and rotated object detectors. Extensive experiments on the MS COCO,
PASCAL VOC, and DOTA benchmarks demonstrate that our method can achieve
considerable AP improvement without any sacrifice on the inference speed. Our
source code and pretrained models are publicly available at
this https URL

本文介绍如何通过局部分区域精细地传递分类和定位知识，实现 logit mimicking 在目标检测领域中的优化，提高模型性能和准确性，理论上证明了定位蒸馏和分类蒸馏具有等效的优化效果。