Knowledge Distillation (KD) transfers the knowledge from a high-capacity
teacher network to strengthen a smaller student. Existing methods focus on
excavating the knowledge hints and transferring the whole knowledge to the
student. However, the knowledge redundancy arises since the knowledge shows
different values to the student at different learning stages. In this paper, we
propose Knowledge Condensation Distillation (KCD). Specifically, the knowledge
value on each sample is dynamically estimated, based on which an
Expectation-Maximization (EM) framework is forged to iteratively condense a
compact knowledge set from the teacher to guide the student learning. Our
approach is easy to build on top of the off-the-shelf KD methods, with no extra
training parameters and negligible computation overhead. Thus, it presents one
new perspective for KD, in which the student that actively identifies teacher's
knowledge in line with its aptitude can learn to learn more effectively and
efficiently. Experiments on standard benchmarks manifest that the proposed KCD
can well boost the performance of student model with even higher distillation
efficiency. Code is available at this https URL

本文提出基于动态学习的知识凝聚蒸馏方法 (KCD)，通过期望最大化 (EM) 框架从教师的知识向学生转移紧凑的知识集，以有效增强学生模型的性能和蒸馏效率。