In Knowledge Distillation, the teacher is generally much larger than the
student, making the solution of the teacher likely to be difficult for the
student to learn. To ease the mimicking difficulty, we introduce a triplet
knowledge distillation mechanism named TriKD. Besides teacher and student,
TriKD employs a third role called anchor model. Before distillation begins, the
pre-trained anchor model delimits a subspace within the full solution space of
the target problem. Solutions within the subspace are expected to be easy
targets that the student could mimic well. Distillation then begins in an
online manner, and the teacher is only allowed to express solutions within the
aforementioned subspace. Surprisingly, benefiting from accurate but
easy-to-mimic hints, the student can finally perform well. After the student is
well trained, it can be used as the new anchor for new students, forming a
curriculum learning strategy. Our experiments on image classification and face
recognition with various models clearly demonstrate the effectiveness of our
method. Furthermore, the proposed TriKD is also effective in dealing with the
overfitting issue. Moreover, our theoretical analysis supports the rationality
of our triplet distillation.

提出了一种三元知识蒸馏 (TriKD) 机制，通过引入锚模型、在线知识传递和课程学习策略，可以有效地教授知识。该方法应用于图像分类和人脸识别中，结果表明了其有效性。