Self-supervised learning (SSL) has made remarkable progress in visual representation learning. Some studies combine SSL with knowledge distillation (SSL-KD) to boost the representation learning performance of small models. In this study, we propose a Multi-mode Online Knowledge Distillation method (MOKD) to boost self-supervised visual representation learning. Different from existing SSL-KD methods that transfer knowledge from a static pre-trained teacher to a student, in MOKD, two different models learn collaboratively in a self-supervised manner. Specifically, MOKD consists of two distillation modes: self-distillation and cross-distillation modes. Among them, self-distillation performs self-supervised learning for each model independently, while cross-distillation realizes knowledge interaction between different models. In cross-distillation, a cross-attention feature search strategy is proposed to enhance the semantic feature alignment between different models. As a result, the two models can absorb knowledge from each other to boost their representation learning performance. Extensive experimental results on different backbones and datasets demonstrate that two heterogeneous models can benefit from MOKD and outperform their independently trained baseline. In addition, MOKD also outperforms existing SSL-KD methods for both the student and teacher models.

本文提出了一种多模在线知识蒸馏方法来提高自监督视觉表征学习，通过自蒸馏和交叉蒸馏两种蒸馏模式，实现了不同模型之间的知识交互，并通过跨注意力特征搜索策略增强了不同模型之间的语义特征对齐，实现了两个异构模型相互吸收知识以提高其表征学习性能。实验证明，这种方法在不同的骨干网络和数据集上都具有较好的性能表现，并且优于现有的 SSL-KD 方法。

自监督视觉表示学习的多模式在线知识蒸馏