The knowledge distillation uses a high-performance teacher network to guide
the student network. However, the performance gap between the teacher and
student networks can affect the student's training. This paper proposes a novel
knowledge distillation algorithm based on dynamic entropy correction to reduce
the gap by adjusting the student instead of the teacher. Firstly, the effect of
changing the output entropy (short for output information entropy) in the
student on the distillation loss is analyzed in theory. This paper shows that
correcting the output entropy can reduce the gap. Then, a knowledge
distillation algorithm based on dynamic entropy correction is created, which
can correct the output entropy in real-time with an entropy controller updated
dynamically by the distillation loss. The proposed algorithm is validated on
the CIFAR100 and ImageNet. The comparison with various state-of-the-art
distillation algorithms shows impressive results, especially in the experiment
on the CIFAR100 regarding teacher-student pair resnet32x4-resnet8x4. The
proposed algorithm raises 2.64 points over the traditional distillation
algorithm and 0.87 points over the state-of-the-art algorithm CRD in
classification accuracy, demonstrating its effectiveness and efficiency.

本文提出了一种基于动态熵校正的知识蒸馏算法，旨在通过调整学生网络而非教师网络来减少它们之间的性能差异。该算法在 CIFAR100 和 ImageNet 数据集上获得了令人印象深刻的结果，特别是在 teacher-resnet32x4 - student-resnet8x4 模型上，该算法在分类准确率上超过了传统算法和当前业内领先的算法 CRD。