Vision Transformers have been rapidly uprising in computer vision thanks to
their outstanding scaling trends, and gradually replacing convolutional neural
networks (CNNs). Recent works on self-supervised learning (SSL) introduce
siamese pre-training tasks, on which Transformer backbones continue to
demonstrate ever stronger results than CNNs. People come to believe that
Transformers or self-attention modules are inherently more suitable than CNNs
in the context of SSL. However, it is noteworthy that most if not all prior
arts of SSL with CNNs chose the standard ResNets as their backbones, whose
architecture effectiveness is known to already lag behind advanced Vision
Transformers. Therefore, it remains unclear whether the self-attention
operation is crucial for the recent advances in SSL - or CNNs can deliver the
same excellence with more advanced designs, too? Can we close the SSL
performance gap between Transformers and CNNs? To answer these intriguing
questions, we apply self-supervised pre-training to the recently proposed,
stronger lager-kernel CNN architecture and conduct an apple-to-apple comparison
with Transformers, in their SSL performance. Our results show that we are able
to build pure CNN SSL architectures that perform on par with or better than the
best SSL-trained Transformers, by just scaling up convolutional kernel sizes
besides other small tweaks. Impressively, when transferring to the downstream
tasks \texttt{MS COCO} detection and segmentation, our SSL pre-trained CNN
model (trained in 100 epochs) achieves the same good performance as the
300-epoch pre-trained Transformer counterpart. We hope this work can help to
better understand what is essential (or not) for self-supervised learning
backbones.

卷积神经网络与视觉 Transformer 在自我监督学习任务中表现相近，通过增加卷积核大小和其他小调整即可达到与 Transformer 相同的性能水平。