Many existing studies on knowledge distillation have focused on methods in
which a student model mimics a teacher model well.
Simply imitating the teacher's knowledge, however, is not sufficient for the
student to surpass that of the teacher.
We explore a method to harness the knowledge of other students to complement
the knowledge of the teacher.
We propose deep collective knowledge distillation for model compression,
called DCKD, which is a method for training student models with rich
information to acquire knowledge from not only their teacher model but also
other student models.
The knowledge collected from several student models consists of a wealth of
information about the correlation between classes.
Our DCKD considers how to increase the correlation knowledge of classes
during training.
Our novel method enables us to create better performing student models for
collecting knowledge.
This simple yet powerful method achieves state-of-the-art performances in
many experiments.
For example, for ImageNet, ResNet18 trained with DCKD achieves 72.27\%, which
outperforms the pretrained ResNet18 by 2.52\%.
For CIFAR-100, the student model of ShuffleNetV1 with DCKD achieves 6.55\%
higher top-1 accuracy than the pretrained ShuffleNetV1.

DCKD 是一种深度集体知识蒸馏的模型压缩方法，旨在通过丰富的信息让学生模型从老师模型和其他学生模型中获取知识，本文探讨了如何在训练过程中提高类别之间的相关性，实验结果表明该方法在 ImageNet 和 CIFAR-100 数据集上取得了最先进的性能表现。