Knowledge distillation (KD) is one of the most potent ways for model
compression. The key idea is to transfer the knowledge from a deep teacher
model (T) to a shallower student (S). However, existing methods suffer from
performance degradation due to the substantial gap between the learning
capacities of S and T. To remedy this problem, this work proposes Residual
Knowledge Distillation (RKD), which further distills the knowledge by
introducing an assistant (A). Specifically, S is trained to mimic the feature
maps of T, and A aids this process by learning the residual error between them.
In this way, S and A complement with each other to get better knowledge from T.
Furthermore, we devise an effective method to derive S and A from a given model
without increasing the total computational cost. Extensive experiments show
that our approach achieves appealing results on popular classification
datasets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods.

本研究提出了一种名为 Residual Knowledge Distillation (RKD) 的知识蒸馏方法，通过引入辅助器来进一步提炼知识，从而解决现有方法由于学习容量间的巨大差距而导致的性能下降问题，并在 CIFAR-100 和 ImageNet 等流行分类数据集上取得优异的成果，超过了现有方法的最新水平。