Existing vision-language pre-training (VLP) methods primarily rely on paired
image-text datasets, which are either annotated by enormous human labors, or
crawled from the internet followed by elaborate data cleaning techniques. To
reduce the dependency on well-aligned image-text pairs, it is promising to
directly leverage the large-scale text-only and image-only corpora. This paper
proposes a data augmentation method, namely cross-modal CutMix (CMC), for
implicit cross-modal alignment learning in unpaired VLP. Specifically, CMC
transforms natural sentences from the textual view into a multi-modal view,
where visually-grounded words in a sentence are randomly replaced by diverse
image patches with similar semantics. There are several appealing proprieties
of the proposed CMC. First, it enhances the data diversity while keeping the
semantic meaning intact for tackling problems where the aligned data are
scarce; Second, by attaching cross-modal noise on uni-modal data, it guides
models to learn token-level interactions across modalities for better
denoising. Furthermore, we present a new unpaired VLP method, dubbed as
VLMixer, that integrates CMC with contrastive learning to pull together the
uni-modal and multi-modal views for better instance-level alignments among
different modalities. Extensive experiments on five downstream tasks show that
VLMixer could surpass previous state-of-the-art unpaired VLP methods.

本文提出一种名为跨模态 CutMix（CMC）的数据增强方法，用于隐式的跨模态对齐学习，在没有对齐图像文本对的情况下从纯文本和图像语料库中学习。同时，该论文还提出了一种新的无对齐图像文本预训练方法，名为 VLMixer，通过将 CMC 与对比学习相结合，可以更好地对齐不同视图中的实例。实验结果表明，VLMixer 可以超过以前的无对齐 VLP 方法。