Resource-constrained perception systems such as edge computing and
vision-for-robotics require vision models to be both accurate and lightweight
in computation and memory usage. While knowledge distillation is a proven
strategy to enhance the performance of lightweight classification models, its
application to structured outputs like object detection and instance
segmentation remains a complicated task, due to the variability in outputs and
complex internal network modules involved in the distillation process. In this
paper, we propose a simple yet surprisingly effective sequential approach to
knowledge distillation that progressively transfers the knowledge of a set of
teacher detectors to a given lightweight student. To distill knowledge from a
highly accurate but complex teacher model, we construct a sequence of teachers
to help the student gradually adapt. Our progressive strategy can be easily
combined with existing detection distillation mechanisms to consistently
maximize student performance in various settings. To the best of our knowledge,
we are the first to successfully distill knowledge from Transformer-based
teacher detectors to convolution-based students, and unprecedentedly boost the
performance of ResNet-50 based RetinaNet from 36.5% to 42.0% AP and Mask R-CNN
from 38.2% to 42.5% AP on the MS COCO benchmark.

对于计算和内存资源有限的感知系统，我们提出了一种简单但非常有效的序列化方法来通过知识蒸馏来提高轻量级分类模型的性能，并成功将基于 Transformer 的教师检测器的知识转化到基于卷积的学生检测器上，从而显著提升了 MS COCO 基准测试上 RetinaNet 和 Mask R-CNN 的性能。