Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps. In order to study the generalization behavior of a distilled student, we propose a new theoretical framework that leverages supervision complexity: a measure of alignment between teacher-provided supervision and the student's neural tangent kernel. The framework highlights a delicate interplay among the teacher's accuracy, the student's margin with respect to the teacher predictions, and the complexity of the teacher predictions. Specifically, it provides a rigorous justification for the utility of various techniques that are prevalent in the context of distillation, such as early stopping and temperature scaling. Our analysis further suggests the use of online distillation, where a student receives increasingly more complex supervision from teachers in different stages of their training. We demonstrate efficacy of online distillation and validate the theoretical findings on a range of image classification benchmarks and model architectures.

本文提出了一个新的理论框架，探究了知识蒸馏的学生的普适性行为，并评估了在线蒸馏的效力。该框架突出了教师提供监督和学生的神经切线核之间的对齐度的复杂性之间微妙的相互作用，为蒸馏中流行的各种技术的效用提供了严密的理论基础。

监督复杂度及其在知识蒸馏中的作用