Understanding self-supervised learning is important but challenging. Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes. However, the recent work of Saunshi et al. argues that the model architecture -- a component largely ignored by previous works -- also has significant influences on the downstream performance of self-supervised learning. In this work, we provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class. In particular, we focus on contrastive learning -- a popular self-supervised learning method that is widely used in the vision domain. We show that when the model has limited capacity, contrastive representations would recover certain special clustering structures that are compatible with the model architecture, but ignore many other clustering structures in the data distribution. As a result, our theory can capture the more realistic setting where contrastive representations have much lower dimensionality than the number of clusters in the data distribution. We instantiate our theory on several synthetic data distributions, and provide empirical evidence to support the theory.

本研究针对自监督学习提供了首个理论分析，其中包括来自模型类祖产的归纳偏差的影响。我们特别关注对比学习 - 一种在视觉领域广泛使用的自监督学习方法。我们发现，当模型具有有限的容量时，对比表示将恢复与模型结构兼容的某些特殊聚类结构，但忽略数据分布中的许多其他聚类结构，从而捕捉了更加现实的情景。我们将理论实例化为几个合成数据分布，并提供实证证据来支持该理论。

对比学习中归纳偏置的理论研究