Multilingual vision-language (V&L) pre-training has achieved remarkable
progress in learning universal representations across different modalities and
languages. In spite of recent success, there still remain challenges limiting
further improvements of V&L pre-trained models in multilingual settings.
Particularly, current V&L pre-training methods rely heavily on strictly-aligned
multilingual image-text pairs generated from English-centric datasets through
machine translation. However, the cost of collecting and translating such
strictly-aligned datasets is usually unbearable. In this paper, we propose
Regularized Contrastive Cross-lingual Cross-modal (RC^3) pre-training, which
further exploits more abundant weakly-aligned multilingual image-text pairs.
Specifically, we design a regularized cross-lingual visio-textual contrastive
learning objective that constrains the representation proximity of
weakly-aligned visio-textual inputs according to textual relevance. Besides,
existing V&L pre-training approaches mainly deal with visual inputs by either
region-of-interest (ROI) features or patch embeddings. We flexibly integrate
the two forms of visual features into our model for pre-training and downstream
multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks
across 6 languages demonstrate the effectiveness of our proposed method over
competitive contrast models with stronger zero-shot capability.

该研究提出了 RC^3 pre-training 方法，该方法利用弱对齐的多语言图像 - 文本对进行跨语言、跨模态的视觉 - 语言预训练，并在下游的多模态任务中表现出更强的效果。