Multi-modal contrastive learning (MMCL) has recently garnered considerable
interest due to its superior performance in visual tasks, achieved by embedding
multi-modal data, such as visual-language pairs. However, there still lack
theoretical understandings of how MMCL extracts useful visual representation
from multi-modal pairs, and particularly, how MMCL outperforms previous
approaches like self-supervised contrastive learning (SSCL). In this paper, by
drawing an intrinsic connection between MMCL and asymmetric matrix
factorization, we establish the first generalization guarantees of MMCL for
visual downstream tasks. Based on this framework, we further unify MMCL and
SSCL by showing that MMCL implicitly performs SSCL with (pseudo) positive pairs
induced by text pairs. Through this unified perspective, we characterize the
advantage of MMCL by showing that text pairs induce more semantically
consistent and diverse positive pairs, which, according to our analysis,
provably benefit downstream generalization. Inspired by this finding, we
propose CLIP-guided resampling methods to significantly improve the downstream
performance of SSCL on ImageNet by leveraging multi-modal information. Code is
available at this https URL

本文通过建立多模态对称矩阵分解和对称矩阵分解的内在联系，证明了多模态对比学习在图像下游任务中的泛化保证，并通过统一模型表明多模态对比学习隐含地通过文本对生成了伪正对，从而具有更多的语义一致和多样性正对，可显著提高基于自我监督的对比学习在图像分类中的性能。