With the improvement of AI chips (e.g., GPU, TPU, and NPU) and the fast
development of the Internet of Things (IoT), some robust deep neural networks
(DNNs) are usually composed of millions or even hundreds of millions of
parameters. Such a large model may not be suitable for directly deploying on
low computation and low capacity units (e.g., edge devices). Knowledge
distillation (KD) has recently been recognized as a powerful model compression
method to decrease the model parameters effectively. The central concept of KD
is to extract useful information from the feature maps of a large model (i.e.,
teacher model) as a reference to successfully train a small model (i.e.,
student model) in which the model size is much smaller than the teacher one.
Although many KD methods have been proposed to utilize the information from the
feature maps of intermediate layers in the teacher model, most did not consider
the similarity of feature maps between the teacher model and the student model.
As a result, it may make the student model learn useless information. Inspired
by the attention mechanism, we propose a novel KD method called representative
teacher key (RTK) that not only considers the similarity of feature maps but
also filters out the useless information to improve the performance of the
target student model. In the experiments, we validate our proposed method with
several backbone networks (e.g., ResNet and WideResNet) and datasets (e.g.,
CIFAR10, CIFAR100, SVHN, and CINIC10). The results show that our proposed RTK
can effectively improve the classification accuracy of the state-of-the-art
attention-based KD method.

本文提出了一种基于注意力机制的知识蒸馏方法，名为代表性教师关键 (point)（RTK），可以实现从大型模型中提取特征信息并训练小型模型，在保证特征相似性的同时过滤掉无用信息，在几个常用的数据集上进行实验，取得了显著的分类准确率提升。