Massive web-crawled image-text datasets lay the foundation for recent
progress in multimodal learning. These datasets are designed with the goal of
training a model to do well on standard computer vision benchmarks, many of
which, however, have been shown to be English-centric (e.g., ImageNet).
Consequently, existing data curation techniques gravitate towards using
predominantly English image-text pairs and discard many potentially useful
non-English samples. Our work questions this practice. Multilingual data is
inherently enriching not only because it provides a gateway to learn about
culturally salient concepts, but also because it depicts common concepts
differently from monolingual data. We thus conduct a systematic study to
explore the performance benefits of using more samples of non-English origins
with respect to English vision tasks. By translating all multilingual
image-text pairs from a raw web crawl to English and re-filtering them, we
increase the prevalence of (translated) multilingual data in the resulting
training set. Pre-training on this dataset outperforms using English-only or
English-dominated datasets on ImageNet, ImageNet distribution shifts,
image-English-text retrieval and on average across 38 tasks from the DataComp
benchmark. On a geographically diverse task like GeoDE, we also observe
improvements across all regions, with the biggest gain coming from Africa. In
addition, we quantitatively show that English and non-English data are
significantly different in both image and (translated) text space. We hope that
our findings motivate future work to be more intentional about including
multicultural and multilingual data, not just when non-English or
geographically diverse tasks are involved, but to enhance model capabilities at
large.

使用多语言数据集进行预训练可以提高在多个视觉任务中的性能，包括对非英语数据的使用，以及增加地理多样性任务中非洲地区的表现。