Multi-modal pre-training models have been intensively explored to bridge
vision and language in recent years. However, most of them explicitly model the
cross-modal interaction between image-text pairs, by assuming that there exists
strong semantic correlation between the text and image modalities. Since this
strong assumption is often invalid in real-world scenarios, we choose to
implicitly model the cross-modal correlation for large-scale multi-modal
pre-training, which is the focus of the Chinese project `WenLan' led by our
team. Specifically, with the weak correlation assumption over image-text pairs,
we propose a two-tower pre-training model called BriVL within the cross-modal
contrastive learning framework. Unlike OpenAI CLIP that adopts a simple
contrastive learning method, we devise a more advanced algorithm by adapting
the latest method MoCo into the cross-modal scenario. By building a large
queue-based dictionary, our BriVL can incorporate more negative samples in
limited GPU resources. We further construct a large Chinese multi-source
image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model.
Extensive experiments demonstrate that the pre-trained BriVL model outperforms
both UNITER and OpenAI CLIP on various downstream tasks.

本文介绍了作者团队领导的 “文兰” 项目的主要研究方向，即通过两个塔的 BriVL 预训练模型和跨模态对比学习框架的先进算法，隐式地建模跨模态关联，从而更成功地实现大规模多模态预训练。同时，他们还建立了一个大规模的中文多源图像文本语料库，称为 RUC-CAS-WenLan，用于 BriVL 模型的预训练。实验结果表明，预训练的 BriVL 模型在各种下游任务中的性能优于 UNITER 和 OpenAI CLIP。