We propose a novel theoretical framework to understand contrastive
self-supervised learning (SSL) methods that employ dual pairs of deep ReLU
networks (e.g., SimCLR). First, we prove that in each SGD update of SimCLR with
various loss functions, including simple contrastive loss, soft Triplet loss
and InfoNCE loss, the weights at each layer are updated by a \emph{covariance
operator} that specifically amplifies initial random selectivities that vary
across data samples but survive averages over data augmentations. To further
study what role the covariance operator plays and which features are learned in
such a process, we model data generation and augmentation processes through a
\emph{hierarchical latent tree model} (HLTM) and prove that the hidden neurons
of deep ReLU networks can learn the latent variables in HLTM, despite the fact
that the network receives \emph{no direct supervision} from these unobserved
latent variables. This leads to a provable emergence of hierarchical features
through the amplification of initially random selectivities through contrastive
SSL. Extensive numerical studies justify our theoretical findings. Code is
released in this https URL

提出了一种新的理论框架来理解对比自监督学习方法，该方法采用双重 ReLU 网络（例如 SimCLR）。我们证明了每个 SGD 更新中的权重是由协方差算子更新的，并进一步说明了协方差算子的作用和应用。通过建立层次潜变量树模型并证明深度 ReLU 网络的隐藏层学习了该模型的潜变量，我们总结了通过对比自监督学习的初始随机选择法放大而形成的层次化特征。