It has been recently demonstrated that multi-generational self-distillation
can improve generalization. Despite this intriguing observation, reasons for
the enhancement remain poorly understood. In this paper, we first demonstrate
experimentally that the improved performance of multi-generational
self-distillation is in part associated with the increasing diversity in
teacher predictions. With this in mind, we offer a new interpretation for
teacher-student training as amortized MAP estimation, such that teacher
predictions enable instance-specific regularization. Our framework allows us to
theoretically relate self-distillation to label smoothing, a commonly used
technique that regularizes predictive uncertainty, and suggests the importance
of predictive diversity in addition to predictive uncertainty. We present
experimental results using multiple datasets and neural network architectures
that, overall, demonstrate the utility of predictive diversity. Finally, we
propose a novel instance-specific label smoothing technique that promotes
predictive diversity without the need for a separately trained teacher model.
We provide an empirical evaluation of the proposed method, which, we find,
often outperforms classical label smoothing.

通过实验证明，多代自蒸馏能够在一定程度上提高模型的性能，之所以会出现这种情况，部分原因在于教学者预测的多样性。我们提出了一种新的教师 - 学员训练解释，即估计自适应正则化，这使得标签平滑的预测不确定性和预测多样性同样重要。我们还提出了一种新的实例特定标签平滑方法，并提供了实验证明其在许多情况下优于传统标签平滑方法。