In knowledge distillation, a student model is trained with supervisions from
both knowledge from a teacher and observations drawn from a training data
distribution. Knowledge of a teacher is considered a subject that holds
inter-class relations which send a meaningful supervision to a student; hence,
much effort has been put to find such knowledge to be distilled. In this paper,
we explore a question that has been given little attention: "when to distill
such knowledge." The question is answered in our work with the concept of model
calibration; we view a teacher model not only as a source of knowledge but also
as a gauge to detect miscalibration of a student. This simple and yet novel
view leads to a hard gate knowledge distillation scheme that switches between
learning from a teacher model and training data. We verify the gating mechanism
in the context of natural language generation at both the token-level and the
sentence-level. Empirical comparisons with strong baselines show that hard gate
knowledge distillation not only improves model generalization, but also
significantly lowers model calibration error.

在知识蒸馏中，我们探索了一个很少被关注的问题：什么时候蒸馏知识？我们提出了模型校准的概念，将教师模型视为不仅提供知识的源头，而且作为检测学生模型失调的一个标尺。这种新颖的观点导致了一个硬门控知识蒸馏方案，可以在教师模型和训练数据之间进行学习，并在自然语言生成的上下文中进行了检验。实证比较表明，硬门控知识蒸馏不仅提高了模型的泛化性能，而且显著降低了模型的校准误差。