While Knowledge Distillation (KD) has been recognized as a useful tool in
many visual tasks, such as supervised classification and self-supervised
representation learning, the main drawback of a vanilla KD framework is its
mechanism, which consumes the majority of the computational overhead on
forwarding through the giant teacher networks, making the entire learning
procedure inefficient and costly. ReLabel, a recently proposed solution,
suggests creating a label map for the entire image. During training, it
receives the cropped region-level label by RoI aligning on a pre-generated
entire label map, allowing for efficient supervision generation without having
to pass through the teachers many times. However, as the KD teachers are from
conventional multi-crop training, there are various mismatches between the
global label-map and region-level label in this technique, resulting in
performance deterioration. In this study, we present a Fast Knowledge
Distillation (FKD) framework that replicates the distillation training phase
and generates soft labels using the multi-crop KD approach, while training
faster than ReLabel since no post-processes such as RoI align and softmax
operations are used. When conducting multi-crop in the same image for data
loading, our FKD is even more efficient than the traditional image
classification framework. On ImageNet-1K, we obtain 79.8% with ResNet-50,
outperforming ReLabel by ~1.0% while being faster. On the self-supervised
learning task, we also show that FKD has an efficiency advantage. Our project
page: this http URL, source code and models
are available at: this https URL

本研究提出了一种快速知识蒸馏 (FKD) 框架来解决知识蒸馏中效率低下的问题，并通过在 ImageNet-1K 和自监督学习任务上的实验展示了其优越性。