Multi-modal contrastive learning (MMCL) has recently garnered considerable
interest due to its superior performance in visual tasks, achieved by embedding
multi-modal data, such as visual-language pairs. However, there still lack
theoretical understandings of how MMCL extracts useful visual representation
from multi-modal pairs, and particularly, how MMCL outperforms previous
approaches like self-supervised contrastive learning (SSCL). In this paper, by
drawing an intrinsic connection between MMCL and asymmetric matrix
factorization, we establish the first generalization guarantees of MMCL for
visual downstream tasks. Based on this framework, we further unify MMCL and
SSCL by showing that MMCL implicitly performs SSCL with (pseudo) positive pairs
induced by text pairs. Through this unified perspective, we characterize the
advantage of MMCL by showing that text pairs induce more semantically
consistent and diverse positive pairs, which, according to our analysis,
provably benefit downstream generalization. Inspired by this finding, we
propose CLIP-guided resampling methods to significantly improve the downstream
performance of SSCL on ImageNet by leveraging multi-modal information. Code is
available at this https URL

本文通过建立多模态对称矩阵分解和对称矩阵分解的内在联系，证明了多模态对比学习在图像下游任务中的泛化保证，并通过统一模型表明多模态对比学习隐含地通过文本对生成了伪正对，从而具有更多的语义一致和多样性正对，可显著提高基于自我监督的对比学习在图像分类中的性能。

多模态对比学习的泛化性

On the Generalization of Multi-modal Contrastive Learning

We consider alternating gradient descent (AGD) with fixed step size $\eta >
0$, applied to the asymmetric matrix factorization objective. We show that, for
a rank-$r$ matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$, $T = \left(
\left(\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}\right)^2
\log(1/\epsilon)\right)$ iterations of alternating gradient descent suffice to
reach an $\epsilon$-optimal factorization $\| \mathbf{A} -
\mathbf{X}_T^{\vphantom{\intercal}} \mathbf{Y}_T^{\intercal} \|_{\rm F}^2 \leq
\epsilon \| \mathbf{A} \|_{\rm F}^2$ with high probability starting from an
atypical random initialization. The factors have rank $d>r$ so that
$\mathbf{X}_T\in\mathbb{R}^{m \times d}$ and $\mathbf{Y}_T \in\mathbb{R}^{n
\times d}$. Experiments suggest that our proposed initialization is not merely
of theoretical benefit, but rather significantly improves convergence of
gradient descent in practice. Our proof is conceptually simple: a uniform
PL-inequality and uniform Lipschitz smoothness constant are guaranteed for a
sufficient number of iterations, starting from our random initialization. Our
proof method should be useful for extending and simplifying convergence
analyses for a broader class of nonconvex low-rank factorization problems.

本文研究了交替梯度下降算法应用于非对称矩阵分解目标函数的收敛性分析，证明了在充分迭代步数内，随机初始化下可以收敛到较优解，此结果可以为更广泛的非凸低秩矩阵分解问题的收敛分析提供帮助，并在实验中得到了验证。