The improvement in the performance of efficient and lightweight models (i.e.,
the student model) is achieved through knowledge distillation (KD), which
involves transferring knowledge from more complex models (i.e., the teacher
model). However, most existing KD techniques rely on Kullback-Leibler (KL)
divergence, which has certain limitations. First, if the teacher distribution
has high entropy, the KL divergence's mode-averaging nature hinders the
transfer of sufficient target information. Second, when the teacher
distribution has low entropy, the KL divergence tends to excessively focus on
specific modes, which fails to convey an abundant amount of valuable knowledge
to the student. Consequently, when dealing with datasets that contain numerous
confounding or challenging samples, student models may struggle to acquire
sufficient knowledge, resulting in subpar performance. Furthermore, in previous
KD approaches, we observed that data augmentation, a technique aimed at
enhancing a model's generalization, can have an adverse impact. Therefore, we
propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages
correlation distance and network pruning. This approach enables KD to
effectively incorporate data augmentation for performance improvement.
Extensive experiments on various datasets, including CIFAR-100, FGVR,
TinyImagenet, and ImageNet, demonstrate our method's superiority over current
state-of-the-art methods.

我们提出了一种利用关联距离和网络修剪的强化稳健知识蒸馏（R2KD）方法，能够有效结合数据增强以提高模型的性能。在各种数据集上进行了广泛的实验，包括 CIFAR-100、FGVR、TinyImagenet 和 ImageNet，这些实验证明了我们的方法优于当前最先进的方法。