Self-supervised Multi-modal Contrastive Learning (SMCL) remarkably advances
modern Vision-Language Pre-training (VLP) models by aligning visual and
linguistic modalities. Due to noises in web-harvested text-image pairs,
however, scaling up training data volume in SMCL presents considerable
obstacles in terms of computational cost and data inefficiency. To improve data
efficiency in VLP, we propose Text-aware Image Mixing (TiMix), which integrates
mix-based data augmentation techniques into SMCL, yielding significant
performance improvements without significantly increasing computational
overhead. We provide a theoretical analysis of TiMixfrom a mutual information
(MI) perspective, showing that mixed data samples for cross-modal contrastive
learning implicitly serve as a regularizer for the contrastive loss. The
experimental results demonstrate that TiMix exhibits a comparable performance
on downstream tasks, even with a reduced amount of training data and shorter
training time, when benchmarked against existing methods. This work empirically
and theoretically demonstrates the potential of data mixing for data-efficient
and computationally viable VLP, benefiting broader VLP model adoption in
practical scenarios.

Self-supervised Multi-modal Contrastive Learning (SMCL) 模型通过整合视觉和语言模态，使现代视觉 - 语言预训练（VLP）模型在无监督学习方面取得显著的进展。然而，由于网络采集的文本 - 图像对中存在噪声问题，在 SMCL 模型中增加训练数据量会带来相当大的计算成本和数据效率问题。为了提高 VLP 模型的数据效率，我们提出了 Text-aware Image Mixing（TiMix）方法，将基于混合的数据增强技术应用到 SMCL 模型中，能够在显著提升性能的同时，不会显著增加计算开销。我们从互信息（MI）的角度对 TiMix 进行了理论分析，表明混合的数据样本能够隐式地作为对比损失的正则化器。实验结果表明，与现有方法相比，即使减少了训练数据量并缩短了训练时间，TiMix 在下游任务上表现出了可比较的性能。该研究从实证和理论上证明了数据混合在数据高效和计算可行的 VLP 中的潜力，为 VLP 模型在实际场景中的广泛应用带来了好处。