Most vision-and-language pretraining research focuses on English tasks.
However, the creation of multilingual multimodal evaluation datasets (e.g.
Multi30K, xGQA, XVNLI, and MaRVL) poses a new challenge in finding high-quality
training data that is both multilingual and multimodal. In this paper, we
investigate whether machine translating English multimodal data can be an
effective proxy for the lack of readily available multilingual data. We call
this framework TD-MML: Translated Data for Multilingual Multimodal Learning,
and it can be applied to any multimodal dataset and model. We apply it to both
pretraining and fine-tuning data with a state-of-the-art model. In order to
prevent models from learning from low-quality translated text, we propose two
metrics for automatically removing such translations from the resulting
datasets. In experiments on five tasks across 20 languages in the IGLUE
benchmark, we show that translated data can provide a useful signal for
multilingual multimodal learning, both at pretraining and fine-tuning.

研究使用机器翻译英文多模态数据作为缺少多语言数据的高质量数据是否有效，并提出了 TD-MML 框架并在 IGLUE 基准测试中展示了它的有效性。