After a large "teacher" neural network has been trained on labeled data, the
probabilities that the teacher assigns to incorrect classes reveal a lot of
information about the way in which the teacher generalizes. By training a small
"student" model to match these probabilities, it is possible to transfer most
of the generalization ability of the teacher to the student, often producing a
much better small model than directly training the student on the training
data. The transfer works best when there are many possible classes because more
is then revealed about the function learned by the teacher, but in cases where
there are only a few possible classes we show that we can improve the transfer
by forcing the teacher to divide each class into many subclasses that it
invents during the supervised training. The student is then trained to match
the subclass probabilities. For datasets where there are known, natural
subclasses we demonstrate that the teacher learns similar subclasses and these
improve distillation. For clickthrough datasets where the subclasses are
unknown we demonstrate that subclass distillation allows the student to learn
faster and better.

通过用小的 “学生” 模型来匹配正确类别的概率，将大 “教师” 神经网络的大部分泛化能力转移到小模型上，训练教师将每个类别分成多个子类别可改进转移，对于有已知自然子类别及未知子类别的数据集，子类别蒸馏可以使学生更快速、更准确地学习。