While self-supervised representation learning (SSL) has received widespread
attention from the community, recent research argue that its performance will
suffer a cliff fall when the model size decreases. The current method mainly
relies on contrastive learning to train the network and in this work, we
propose a simple yet effective Distilled Contrastive Learning (DisCo) to ease
the issue by a large margin. Specifically, we find the final embedding obtained
by the mainstream SSL methods contains the most fruitful information, and
propose to distill the final embedding to maximally transmit a teacher's
knowledge to a lightweight model by constraining the last embedding of the
student to be consistent with that of the teacher. In addition, in the
experiment, we find that there exists a phenomenon termed Distilling BottleNeck
and present to enlarge the embedding dimension to alleviate this problem. Our
method does not introduce any extra parameter to lightweight models during
deployment. Experimental results demonstrate that our method achieves the
state-of-the-art on all lightweight models. Particularly, when
ResNet-101/ResNet-50 is used as teacher to teach EfficientNet-B0, the linear
result of EfficientNet-B0 on ImageNet is very close to ResNet-101/ResNet-50,
but the number of parameters of EfficientNet-B0 is only 9.4\%/16.3\% of
ResNet-101/ResNet-50. Code is available at this https URL
com/Yuting-Gao/DisCo-pytorch.

本文提出了一种名为 Distilled Contrastive Learning (DisCo) 的方法，通过将主流自监督学习方法中蕴含的最有价值的信息提取出来，然后通过约束学生模型的最终嵌入与教师模型的嵌入保持一致，从而大幅缓解模型规模减小时的性能退化问题，实验结果表明，DisCo 在所有轻型模型上取得了最先进的性能，并且在部署过程中不会引入任何额外的参数。