Self-distillation (SD) is the process of training a student model using the outputs of a teacher model, with both models sharing the same architecture. Our study theoretically examines SD in multi-class classification with cross-entropy loss, exploring both multi-round SD and SD with refined teacher outputs, inspired by partial label learning (PLL). By deriving a closed-form solution for the student model's outputs, we discover that SD essentially functions as label averaging among instances with high feature correlations. Initially beneficial, this averaging helps the model focus on feature clusters correlated with a given instance for predicting the label. However, it leads to diminishing performance with increasing distillation rounds. Additionally, we demonstrate SD's effectiveness in label noise scenarios and identify the label corruption condition and minimum number of distillation rounds needed to achieve 100% classification accuracy. Our study also reveals that one-step distillation with refined teacher outputs surpasses the efficacy of multi-step SD using the teacher's direct output in high noise rate regimes.

自蒸馏是使用教师模型的输出来训练学生模型的过程，本研究在多类别分类中从理论上探讨了自蒸馏，研究了多轮自蒸馏和使用精炼教师输出的自蒸馏，其灵感源于部分标签学习（PLL）。我们通过为学生模型的输出推导出一个封闭形式解，发现自蒸馏在具有高特征相关性的实例中起到标签平均化的作用。这种平均化有助于模型关注与给定实例相关的特征聚类以预测标签，但在蒸馏轮次增多时会导致性能衰减。此外，我们证明了自蒸馏在标签噪声情景中的有效性，并确定了达到100%分类准确率所需的标签污染条件和最小蒸馏轮次。我们的研究还揭示了在高噪声率条件下，一步精炼教师输出的蒸馏超过了使用教师直接输出的多步自蒸馏的效果。

多类别标签噪声下的自我蒸馏和部分标签学习的理解