Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network
to learn knowledge from itself without any guidance from extra networks. This
paper proposes to perform Self-KD from image Mixture (MixSKD), which integrates
these two techniques into a unified framework. MixSKD mutually distills feature
maps and probability distributions between the random pair of original images
and their mixup images in a meaningful way. Therefore, it guides the network to
learn cross-image knowledge by modelling supervisory signals from mixup images.
Moreover, we construct a self-teacher network by aggregating multi-stage
feature maps for providing soft labels to supervise the backbone classifier,
further improving the efficacy of self-boosting. Experiments on image
classification and transfer learning to object detection and semantic
segmentation demonstrate that MixSKD outperforms other state-of-the-art Self-KD
and data augmentation methods. The code is available at
this https URL

本文提出了将 Self-KD 应用于图像混合 (MixSKD) 的方法，该方法能够将原始图像和混合图像之间的特征映射和概率分布相互蒸馏，以便以跨图像的知识指导网络的学习，实验表明该方法优于其他最先进的 Self-KD 和数据增强方法。