Knowledge distillation is a widely applicable technique for training a
student neural network under the guidance of a trained teacher network. For
example, in neural network compression, a high-capacity teacher is distilled to
train a compact student; in privileged learning, a teacher trained with
privileged data is distilled to train a student without access to that data.
The distillation loss determines how a teacher's knowledge is captured and
transferred to the student. In this paper, we propose a new form of knowledge
distillation loss that is inspired by the observation that semantically similar
inputs tend to elicit similar activation patterns in a trained network.
Similarity-preserving knowledge distillation guides the training of a student
network such that input pairs that produce similar (dissimilar) activations in
the teacher network produce similar (dissimilar) activations in the student
network. In contrast to previous distillation methods, the student is not
required to mimic the representation space of the teacher, but rather to
preserve the pairwise similarities in its own representation space. Experiments
on three public datasets demonstrate the potential of our approach.

本文提出了新型的知识蒸馏损失函数，其通过保留教师神经网络中相似输入的激活模式特征，指导学生神经网络的训练，使其在保留各自的表征空间中，能够准确地保持输入的相似度。实验结果表明了该方法的潜力。