Knowledge distillation is an effective transfer of knowledge from a heavy
network (teacher) to a small network (student) to boost students' performance.
Self-knowledge distillation, the special case of knowledge distillation, has
been proposed to remove the large teacher network training process while
preserving the student's performance. This paper introduces a novel
Self-knowledge distillation approach via Siamese representation learning, which
minimizes the difference between two representation vectors of the two
different views from a given sample. Our proposed method, SKD-SRL, utilizes
both soft label distillation and the similarity of representation vectors.
Therefore, SKD-SRL can generate more consistent predictions and representations
in various views of the same data point. Our benchmark has been evaluated on
various standard datasets. The experimental results have shown that SKD-SRL
significantly improves the accuracy compared to existing supervised learning
and knowledge distillation methods regardless of the networks.

本文提出了一种新的自知识蒸馏方法，通过孪生表示学习来最小化给定样本的两个不同视图的表示向量之间的差异，并利用软标签蒸馏和表示向量的相似性，因此能够在同一数据点的各种视图中生成更一致的预测和表示，并在各个标准数据集上验证了其显着提高精确度的效果。