How to train an ideal teacher for knowledge distillation is still an open
problem. It has been widely observed that a teacher minimizing the empirical
risk not necessarily yields the best performing student, suggesting a
fundamental discrepancy between the common practice in teacher network training
and the distillation objective. To fill this gap, we propose a novel
student-oriented teacher network training framework SoTeacher, inspired by
recent findings that student performance hinges on teacher's capability to
approximate the true label distribution of training samples. We theoretically
established that (1) the empirical risk minimizer with proper scoring rules as
loss function can provably approximate the true label distribution of training
data if the hypothesis function is locally Lipschitz continuous around training
samples; and (2) when data augmentation is employed for training, an additional
constraint is required that the minimizer has to produce consistent predictions
across augmented views of the same training input. In light of our theory,
SoTeacher renovates the empirical risk minimization by incorporating Lipschitz
regularization and consistency regularization. It is worth mentioning that
SoTeacher is applicable to almost all teacher-student architecture pairs,
requires no prior knowledge of the student upon teacher's training, and induces
almost no computation overhead. Experiments on two benchmark datasets confirm
that SoTeacher can improve student performance significantly and consistently
across various knowledge distillation algorithms and teacher-student pairs.

提出一种基于学生的教师网络训练框架 SoTeacher，通过引入 Lipschitz 正则化和一致性正则化改善知识蒸馏算法中教师网络的训练方法，实验证明该方法适用于几乎所有的教师 - 学生架构对，并且可以显著、一致地提高学生的性能。