We study the relative effects of data augmentations, pretraining algorithms, and model architectures in Self-Supervised Learning (SSL). While the recent literature in this space leaves the impression that the pretraining algorithm is of critical importance to performance, understanding its effect is complicated by the difficulty in making objective and direct comparisons between methods. We propose a new framework which unifies many seemingly disparate SSL methods into a single shared template. Using this framework, we identify aspects in which methods differ and observe that in addition to changing the pretraining algorithm, many works also use new data augmentations or more powerful model architectures. We compare several popular SSL methods using our framework and find that many algorithmic additions, such as prediction networks or new losses, have a minor impact on downstream task performance (often less than $1\%$), while enhanced augmentation techniques offer more significant performance improvements ($2-4\%$). Our findings challenge the premise that SSL is being driven primarily by algorithmic improvements, and suggest instead a bitter lesson for SSL: that augmentation diversity and data / model scale are more critical contributors to recent advances in self-supervised learning.

我们研究了自监督学习中数据增强、预训练算法和模型架构的相对效果。通过比较多个著名的自监督学习方法，我们发现增强技术对性能的改进更为显著，而许多算法改进只有微小的影响。我们的发现挑战了自监督学习主要由算法改进主导的预设，并指出多样的增强方法以及数据和模型规模是自监督学习近期进展的更为关键的因素。

增强方法与算法：自监督学习中的有效途径