Multilingual vision-language (V&L) pre-training has achieved remarkable
progress in learning universal representations across different modalities and
languages. In spite of recent success, there still remain challenges limiting
further improvements of V&L pre-trained models in multilingual settings.
Particularly, current V&L pre-training methods rely heavily on strictly-aligned
multilingual image-text pairs generated from English-centric datasets through
machine translation. However, the cost of collecting and translating such
strictly-aligned datasets is usually unbearable. In this paper, we propose
Regularized Contrastive Cross-lingual Cross-modal (RC^3) pre-training, which
further exploits more abundant weakly-aligned multilingual image-text pairs.
Specifically, we design a regularized cross-lingual visio-textual contrastive
learning objective that constrains the representation proximity of
weakly-aligned visio-textual inputs according to textual relevance. Besides,
existing V&L pre-training approaches mainly deal with visual inputs by either
region-of-interest (ROI) features or patch embeddings. We flexibly integrate
the two forms of visual features into our model for pre-training and downstream
multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks
across 6 languages demonstrate the effectiveness of our proposed method over
competitive contrast models with stronger zero-shot capability.

该研究提出了 RC^3 pre-training 方法，该方法利用弱对齐的多语言图像 - 文本对进行跨语言、跨模态的视觉 - 语言预训练，并在下游的多模态任务中表现出更强的效果。

RC3：正则化对比交叉语言交叉模态预训练

RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training

English-based Vision-Language Pre-training (VLP) has achieved great success
in various downstream tasks. Some efforts have been taken to generalize this
success to non-English languages through Multilingual Vision-Language
Pre-training (M-VLP). However, due to the large number of languages, M-VLP
models often require huge computing resources and cannot be flexibly extended
to new languages. In this work, we propose a \textbf{M}ulti\textbf{L}ingual
\textbf{A}cquisition (MLA) framework that can easily generalize a monolingual
Vision-Language Pre-training model into multilingual. Specifically, we design a
lightweight language acquisition encoder based on state-of-the-art monolingual
VLP models. We further propose a two-stage training strategy to optimize the
language acquisition encoder, namely the Native Language Transfer stage and the
Language Exposure stage. With much less multilingual training data and
computing resources, our model achieves state-of-the-art performance on
multilingual image-text and video-text retrieval benchmarks.

本研究提出了一种 MultiLingual Acquisition（MLA）框架，该框架可以将单语言视觉语言预训练模型轻松推广到多语言环境中，并采用两阶段训练策略来优化语言获取编码器。通过更少的数据和计算资源，我们的模型在多语言图像文本和视频文本检索基准上实现了最先进的性能。