Existing vision-language methods typically support two languages at a time at
most. In this paper, we present a modular approach which can easily be
incorporated into existing vision-language methods in order to support many
languages. We accomplish this by learning a single shared Multimodal Universal
Language Embedding (MULE) which has been visually-semantically aligned across
all languages. Then we learn to relate MULE to visual data as if it were a
single language. Our method is not architecture specific, unlike prior work
which typically learned separate branches for each language, enabling our
approach to easily be adapted to many vision-language methods and tasks. Since
MULE learns a single language branch in the multimodal model, we can also scale
to support many languages, and languages with fewer annotations can take
advantage of the good representation learned from other (more abundant)
language data. We demonstrate the effectiveness of MULE on the bidirectional
image-sentence retrieval task, supporting up to four languages in a single
model. In addition, we show that Machine Translation can be used for data
augmentation in multilingual learning, which, combined with MULE, improves mean
recall by up to 21.9% on a single-language compared to prior work, with the
most significant gains seen on languages with relatively few annotations. Our
code is publicly available.

本文提出了一种模块化的方法，通过学习单一的共享多模态通用语言嵌入（MULE）来支持多种语言，其在多语言图像 - 句子检索任务上的表现得到了证明。此外，我们还展示了机器翻译可以用于多语言学习中的数据增强，再结合 MULE，可以将单一语言的平均召回率提高多达 21.9%，特别是对于有限注释的语言。