Recent research on knowledge distillation has increasingly focused on logit distillation because of its simplicity, effectiveness, and versatility in model compression. In this paper, we introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods. Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions, creating a conflict between the standard distillation loss and the cross-entropy loss. This conflict can undermine the consistency of the student model's learning objectives. Previous attempts to use labels to empirically correct teacher predictions may undermine the class correlation. In contrast, our RLD employs labeling information to dynamically refine teacher logits. In this way, our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations, thus enhancing the value and efficiency of distilled knowledge. Experimental results on CIFAR-100 and ImageNet demonstrate its superiority over existing methods. The code is provided at \text{https://github.com/zju-SWJ/RLD}.

本研究针对现有日志蒸馏方法的局限性，提出了一种新的精炼日志蒸馏(RLD)方法。该方法通过动态精炼教师模型的日志预测，消除误导性信息，保留重要的类别相关性，从而提高学生模型的学习效果。实验结果表明，在CIFAR-100和ImageNet数据集上，RLD相较于其他方法具有更优越的表现。

精炼日志蒸馏的知识蒸馏