Resource-constrained perception systems such as edge computing and
vision-for-robotics require vision models to be both accurate and lightweight
in computation and memory usage. While knowledge distillation is a proven
strategy to enhance the performance of lightweight classification models, its
application to structured outputs like object detection and instance
segmentation remains a complicated task, due to the variability in outputs and
complex internal network modules involved in the distillation process. In this
paper, we propose a simple yet surprisingly effective sequential approach to
knowledge distillation that progressively transfers the knowledge of a set of
teacher detectors to a given lightweight student. To distill knowledge from a
highly accurate but complex teacher model, we construct a sequence of teachers
to help the student gradually adapt. Our progressive strategy can be easily
combined with existing detection distillation mechanisms to consistently
maximize student performance in various settings. To the best of our knowledge,
we are the first to successfully distill knowledge from Transformer-based
teacher detectors to convolution-based students, and unprecedentedly boost the
performance of ResNet-50 based RetinaNet from 36.5% to 42.0% AP and Mask R-CNN
from 38.2% to 42.5% AP on the MS COCO benchmark.

对于计算和内存资源有限的感知系统，我们提出了一种简单但非常有效的序列化方法来通过知识蒸馏来提高轻量级分类模型的性能，并成功将基于 Transformer 的教师检测器的知识转化到基于卷积的学生检测器上，从而显著提升了 MS COCO 基准测试上 RetinaNet 和 Mask R-CNN 的性能。

多教师渐进蒸馏的轻量级目标检测学习

Learning Lightweight Object Detectors via Multi-Teacher Progressive  Distillation

This paper focuses on optimal unimodal transformation of the score outputs of
a univariate learning model under linear loss functions. We demonstrate that
the optimal mapping between score values and the target region is a rectangular
function. To produce this optimal rectangular fit for the observed samples, we
propose a sequential approach that can its estimation with each incoming new
sample. Our approach has logarithmic time complexity per iteration and is
optimally efficient.

本文研究了单变量学习模型的分数输出的最优单峰转换，并证明了分数值和目标区域之间的最优映射是一个矩形函数。我们提出了一种顺序方法来生成观察样本的最优矩形适配，并且该方法可以随着每个新样本的到来进行估计。我们的方法每次迭代的时间复杂度为对数，效率是最优的。