While self-supervised representation learning (SSL) has proved to be
effective in the large model, there is still a huge gap between the SSL and
supervised method in the lightweight model when following the same solution. We
delve into this problem and find that the lightweight model is prone to
collapse in semantic space when simply performing instance-wise contrast. To
address this issue, we propose a relation-wise contrastive paradigm with
Relation Knowledge Distillation (ReKD). We introduce a heterogeneous teacher to
explicitly mine the semantic information and transferring a novel relation
knowledge to the student (lightweight model). The theoretical analysis supports
our main concern about instance-wise contrast and verify the effectiveness of
our relation-wise contrastive learning. Extensive experimental results also
demonstrate that our method achieves significant improvements on multiple
lightweight models. Particularly, the linear evaluation on AlexNet obviously
improves the current state-of-art from 44.7% to 50.1%, which is the first work
to get close to the supervised 50.5%. Code will be made available.

通过引入异构的 teacher 和 relation-wise 对比学习，我们成功地解决了在轻量级模型中进行 instance-wise 对比时在语义空间中容易崩溃的问题，并通过 AlexNet 线性评估验证了该方法的有效性，达到了接近于有监督学习的 50.5% 的水平。