In this paper, we introduce Cross-View Language Modeling, a simple and
effective language model pre-training framework that unifies cross-lingual
cross-modal pre-training with shared architectures and objectives. Our approach
is motivated by a key observation that cross-lingual and cross-modal
pre-training share the same goal of aligning two different views of the same
object into a common semantic space. To this end, the cross-view language
modeling framework considers both multi-modal data (i.e., image-caption pairs)
and multi-lingual data (i.e., parallel sentence pairs) as two different views
of the same object, and trains the model to align the two views by maximizing
the mutual information between them with conditional masked language modeling
and contrastive learning. We pre-train CCLM, a Cross-lingual Cross-modal
Language Model, with the cross-view language modeling framework. Empirical
results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual
image-text retrieval datasets show that while conceptually simpler, CCLM
significantly outperforms the prior state-of-the-art with an average absolute
improvement of over 10%. Notably, CCLM is the first multi-lingual multi-modal
model that surpasses the translate-test performance of representative English
vision-language models by zero-shot cross-lingual transfer.

本研究介绍了一种名为 Cross-View Language Modeling 的语言模型预训练框架，通过共享架构和目标统一交叉语言交叉模态预训练。我们的方法将多模态数据（即图像字幕对）和多语言数据（即平行句子对）作为同一对象的两个不同视角，通过条件掩蔽语言建模和对比学习对两个视角进行对齐。在多语言多模态基准 IGLUE 和两个多语言图像文本检索数据集上的实证结果表明，CCLM 显着优于先前的最新技术，在绝对平均改进方面超过 10％，是第一个在零 - shot 跨语言传输下超越代表性的英语视觉语言模型的多语言多模态模型。