We investigate cross-quality knowledge distillation (CQKD), a knowledge
distillation method where knowledge from a teacher network trained with
full-resolution images is transferred to a student network that takes as input
low-resolution images. As image size is a deciding factor for the computational
load of computer vision applications, CQKD notably reduces the requirements by
only using the student network at inference time. Our experimental results show
that CQKD outperforms supervised learning in large-scale image classification
problems. We also highlight the importance of calibrating neural networks: we
show that with higher temperature smoothing of the teacher's output
distribution, the student distribution exhibits a higher entropy, which leads
to both, a lower calibration error and a higher network accuracy.

本文研究交叉质量知识蒸馏（CQKD），该方法将使用全分辨率图像进行训练的教师网络的知识转移到以低分辨率图像作为输入的学生网络中。CQKD 在大规模图像分类问题上的表现优于监督学习，并强调了神经网络校准的重要性：我们表明，通过更高的温度平滑教师输出分布，学生分布会表现出更高的熵，从而导致更低的校准误差和更高的网络准确性。