Contrastive learning has recently seen tremendous success in self-supervised
learning. So far, however, it is largely unclear why the learned
representations generalize so effectively to a large variety of downstream
tasks. We here prove that feedforward models trained with objectives belonging
to the commonly used InfoNCE family learn to implicitly invert the underlying
generative model of the observed data. While the proofs make certain
statistical assumptions about the generative model, we observe empirically that
our findings hold even if these assumptions are severely violated. Our theory
highlights a fundamental connection between contrastive learning, generative
modeling, and nonlinear independent component analysis, thereby furthering our
understanding of the learned representations as well as providing a theoretical
foundation to derive more effective contrastive losses.

使用 InfoNCE 系列的 feedforward 模型通过隐式反转观察数据的基础生成模型来实现使学习到的表示方法适用于大量下游任务的目标，这种理论强调了对比学习，生成建模和非线性独立成分分析之间的基本联系，为推导出更有效的对比损失提供了理论基础。