Offline distillation is a two-stage pipeline that requires expensive
resources to train a teacher network and then distill the knowledge to a
student for deployment. Online knowledge distillation, on the other hand, is a
one-stage strategy that alleviates the requirement with mutual learning and
collaborative learning. Recent peer collaborative learning (PCL) integrates
online ensemble, collaboration of base networks and temporal mean teacher to
construct effective knowledge. However, the model collapses occasionally in PCL
due to high homogenization between the student and the teacher. In this paper,
the cause of the high homogenization is analyzed and the solution is presented.
A decoupled knowledge for online knowledge distillation is generated by an
independent teacher, separate from the student. Such design can increase the
diversity between the networks and reduce the possibility of model collapse. To
obtain early decoupled knowledge, an initialization scheme for the teacher is
devised, and a 2D geometry-based analysis experiment is conducted under ideal
conditions to showcase the effectiveness of this scheme. Moreover, to improve
the teacher's supervisory resilience, a decaying ensemble scheme is devised. It
assembles the knowledge of the teacher to which a dynamic weight which is large
at the start of the training and gradually decreases with the training process
is assigned. The assembled knowledge serves as a strong teacher during the
early training and the decreased-weight-assembled knowledge can eliminate the
distribution deviation under the potentially overfitted teacher's supervision.
A Monte Carlo-based simulation is conducted to evaluate the convergence.
Extensive experiments on CIFAR-10, CIFAR-100 and TinyImageNet show the
superiority of our method. Ablation studies and further analysis demonstrate
the effectiveness.

通过独立的教师生成在线知识蒸馏的分离知识，以增加网络之间的差异性和降低模型崩溃的可能性，并采用衰减集成方案来提高教师的监督韧性。通过对 CIFAR-10、CIFAR-100 和 TinyImageNet 的广泛实验验证了我们方法的优越性，并进行了消融研究和进一步分析来证明其有效性。