In real teaching scenarios, an excellent teacher always teaches what he (or
she) is good at but the student is not. This gives the student the best
assistance in making up for his (or her) weaknesses and becoming a good one
overall. Enlightened by this, we introduce the "Teaching what you Should Teach"
strategy into a knowledge distillation framework, and propose a data-based
distillation method named "TST" that searches for desirable augmented samples
to assist in distilling more efficiently and rationally. To be specific, we
design a neural network-based data augmentation module with priori bias, which
assists in finding what meets the teacher's strengths but the student's
weaknesses, by learning magnitudes and probabilities to generate suitable data
samples. By training the data augmentation module and the generalized
distillation paradigm in turn, a student model is learned with excellent
generalization ability. To verify the effectiveness of our method, we conducted
extensive comparative experiments on object recognition, detection, and
segmentation tasks. The results on the CIFAR-10, ImageNet-1k, MS-COCO, and
Cityscapes datasets demonstrate that our method achieves state-of-the-art
performance on almost all teacher-student pairs. Furthermore, we conduct
visualization studies to explore what magnitudes and probabilities are needed
for the distillation process.

该研究提出了一种名为 “Teaching what you Should Teach” 的策略，将其引入到知识蒸馏框架中。通过神经网络数据增强模块，该数据驱动蒸馏方法可以在查找适当的数据样本时帮助学生弥补其弱点，让学生成为全面优秀的人。此方法在目标识别、检测和分割任务上，通过详细的比较实验和探索性研究，证明它具有先进的性能和一定的推广能力。