Knowledge distillation (KD) is a technique used to transfer knowledge from an
overparameterized teacher network to a less-parameterized student network,
thereby minimizing the incurred performance loss. KD methods can be categorized
into offline and online approaches. Offline KD leverages a powerful pretrained
teacher network, while online KD allows the teacher network to be adjusted
dynamically to enhance the learning effectiveness of the student network.
Recently, it has been discovered that sharing the classifier of the teacher
network can significantly boost the performance of the student network with
only a minimal increase in the number of network parameters. Building on these
insights, we propose adaptive teaching with a shared classifier (ATSC). In
ATSC, the pretrained teacher network self-adjusts to better align with the
learning needs of the student network based on its capabilities, and the
student network benefits from the shared classifier, enhancing its performance.
Additionally, we extend ATSC to environments with multiple teachers. We conduct
extensive experiments, demonstrating the effectiveness of the proposed KD
method. Our approach achieves state-of-the-art results on the CIFAR-100 and
ImageNet datasets in both single-teacher and multiteacher scenarios, with only
a modest increase in the number of required model parameters. The source code
is publicly available at this https URL

通过共享分类器实现自适应教学的知识蒸馏方法（ATSC），能够在网络参数数量仅略有增加的情况下，显著提升学生网络的性能，并在单教师和多教师场景下，在 CIFAR-100 和 ImageNet 数据集上实现了最先进的结果。