Knowledge distillation (KD) has received much attention due to its success in
compressing networks to allow for their deployment in resource-constrained
systems. While the problem of adversarial robustness has been studied before in
the KD setting, previous works overlook what we term the relative calibration
of the student network with respect to its teacher in terms of soft
confidences. In particular, we focus on two crucial questions with regard to a
teacher-student pair: (i) do the teacher and student disagree at points close
to correctly classified dataset examples, and (ii) is the distilled student as
confident as the teacher around dataset examples? These are critical questions
when considering the deployment of a smaller student network trained from a
robust teacher within a safety-critical setting. To address these questions, we
introduce a faithful imitation framework to discuss the relative calibration of
confidences, as well as provide empirical and certified methods to evaluate the
relative calibration of a student w.r.t. its teacher. Further, to verifiably
align the relative calibration incentives of the student to those of its
teacher, we introduce faithful distillation. Our experiments on the MNIST and
Fashion-MNIST datasets demonstrate the need for such an analysis and the
advantages of the increased verifiability of faithful distillation over
alternative adversarial distillation methods.

本文探讨了知识蒸馏（KD）中的师生对相对可靠性的问题，提出了一种忠实模仿框架并提供了经验和认证方法来评估学生与其老师的相对校准，同时介绍了一种忠实蒸馏方法，其在 MNIST 和 Fashion-MNIST 数据集上的实验表明了其优越性。