Knowledge distillation is one of the methods for model compression, and
existing knowledge distillation techniques focus on how to improve the
distillation algorithm so as to enhance the distillation efficdiency. This
paper introduces dynamic incremental learning into knowledge distillation and
proposes a distillation strategy for education distillation. Specifically, it
is proposed to look at fragmented student models divided from the full student
model as low models. As the grade level rises, fragmented student models deepen
in conjunction with designed teaching reference layers, while learning and
distilling from more teacher models. By moving from lower to higher grades,
fragmented student models were gradually integrated into a complete target
student model, and the performance of the student models gradually improved
from lower to senior grades of the stage. Education distillation strategies
combined with distillation algorithms outperform the results of single
distillation algorithms on the public dataset CIFAR100,Caltech256, Food-101
dataset.

本文将动态增量学习引入到知识蒸馏中，提出了一种教育蒸馏的蒸馏策略，通过将学生模型从完整的学生模型分割为低级模型，结合设计的教学参考层，逐渐提升学生模型的年级，并从更多的教师模型中进行学习和蒸馏，使得学生模型的性能逐渐从低级到高级逐阶段提高。教育蒸馏策略结合蒸馏算法在公共数据集 CIFAR100、Caltech256 和 Food-101 数据集上比单一蒸馏算法取得了更优的结果。