Knowledge Distillation (KD) transfers knowledge from a larger "teacher" model
to a compact "student" model, guiding the student with the "dark knowledge"
$\unicode{x2014}$ the implicit insights present in the teacher's soft
predictions. Although existing KDs have shown the potential of transferring
knowledge, the gap between the two parties still exists. With a series of
investigations, we argue the gap is the result of the student's overconfidence
in prediction, signaling an imbalanced focus on pronounced features while
overlooking the subtle yet crucial dark knowledge. To overcome this, we
introduce the Entropy-Reweighted Knowledge Distillation (ER-KD), a novel
approach that leverages the entropy in the teacher's predictions to reweight
the KD loss on a sample-wise basis. ER-KD precisely refocuses the student on
challenging instances rich in the teacher's nuanced insights while reducing the
emphasis on simpler cases, enabling a more balanced knowledge transfer.
Consequently, ER-KD not only demonstrates compatibility with various
state-of-the-art KD methods but also further enhances their performance at
negligible cost. This approach offers a streamlined and effective strategy to
refine the knowledge transfer process in KD, setting a new paradigm in the
meticulous handling of dark knowledge. Our code is available at
this https URL

利用熵重新加权的知识蒸馏 (ER-KD) 方法通过重新调整学生模型的关注点，减少对简单情况的侧重，从而实现在知识传递过程中更加平衡的效果。