Understanding self-supervised learning is important but challenging. Previous
theoretical works study the role of pretraining losses, and view neural
networks as general black boxes. However, the recent work of Saunshi et al.
argues that the model architecture -- a component largely ignored by previous
works -- also has significant influences on the downstream performance of
self-supervised learning. In this work, we provide the first theoretical
analysis of self-supervised learning that incorporates the effect of inductive
biases originating from the model class. In particular, we focus on contrastive
learning -- a popular self-supervised learning method that is widely used in
the vision domain. We show that when the model has limited capacity,
contrastive representations would recover certain special clustering structures
that are compatible with the model architecture, but ignore many other
clustering structures in the data distribution. As a result, our theory can
capture the more realistic setting where contrastive representations have much
lower dimensionality than the number of clusters in the data distribution. We
instantiate our theory on several synthetic data distributions, and provide
empirical evidence to support the theory.

本研究针对自监督学习提供了首个理论分析，其中包括来自模型类祖产的归纳偏差的影响。我们特别关注对比学习 - 一种在视觉领域广泛使用的自监督学习方法。我们发现，当模型具有有限的容量时，对比表示将恢复与模型结构兼容的某些特殊聚类结构，但忽略数据分布中的许多其他聚类结构，从而捕捉了更加现实的情景。我们将理论实例化为几个合成数据分布，并提供实证证据来支持该理论。

对比学习中归纳偏置的理论研究

A Theoretical Study of Inductive Biases in Contrastive Learning

The recently established RPCA method provides us a convenient way to restore
low-rank matrices from grossly corrupted observations. While elegant in theory
and powerful in reality, RPCA may be not an ultimate solution to the low-rank
matrix recovery problem. Indeed, its performance may not be perfect even when
data are strictly low-rank. This is because conventional RPCA ignores the
clustering structures of the data which are ubiquitous in modern applications.
As the number of cluster grows, the coherence of data keeps increasing, and
accordingly, the recovery performance of RPCA degrades. We show that the
challenges raised by coherent data (i.e., the data with high coherence) could
be alleviated by Low-Rank Representation (LRR), provided that the dictionary in
LRR is configured appropriately. More precisely, we mathematically prove that
if the dictionary itself is low-rank then LRR is immune to the coherence
parameter which increases with the underlying cluster number. This provides an
elementary principle for dealing with coherent data. Subsequently, we devise a
practical algorithm to obtain proper dictionaries in unsupervised environments.
Our extensive experiments on randomly generated matrices verify our claims.

研究表明，当数据具有高相干性时，常规的 RPCA 方法表现不佳。使用低秩表示（LRR）能够缓解这一问题，并且当字典本身是低秩的时，LRR 更能有效地处理相干数据。作者设计了一种无监督环境下获取适当字典的实用算法，并在随机生成的矩阵上进行广泛实验以验证其结果。