Despite the popularity and efficacy of knowledge distillation, there is
limited understanding of why it helps. In order to study the generalization
behavior of a distilled student, we propose a new theoretical framework that
leverages supervision complexity: a measure of alignment between
teacher-provided supervision and the student's neural tangent kernel. The
framework highlights a delicate interplay among the teacher's accuracy, the
student's margin with respect to the teacher predictions, and the complexity of
the teacher predictions. Specifically, it provides a rigorous justification for
the utility of various techniques that are prevalent in the context of
distillation, such as early stopping and temperature scaling. Our analysis
further suggests the use of online distillation, where a student receives
increasingly more complex supervision from teachers in different stages of
their training. We demonstrate efficacy of online distillation and validate the
theoretical findings on a range of image classification benchmarks and model
architectures.

本文提出了一个新的理论框架，探究了知识蒸馏的学生的普适性行为，并评估了在线蒸馏的效力。该框架突出了教师提供监督和学生的神经切线核之间的对齐度的复杂性之间微妙的相互作用，为蒸馏中流行的各种技术的效用提供了严密的理论基础。