A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements -- cross-fitting and loss correction -- to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.

使用半参数推断方法将知识蒸馏转换为目标学生模型、未知贝叶斯类概率和教师概率的plug-in估计值，引入交叉适应和损失校正两种方式来改善教师过度拟合和欠拟合对学生性能的影响，为标准蒸馏的预测误差提供了新的保证，并在表格式和图像数据上进行实证验证，观察到与知识蒸馏增强相关的一致改进。

知识蒸馏作为半参数推断