Multimodal Large Language Models (mLLMs) are trained on a large amount of
text-image data. While most mLLMs are trained on caption-like data only,
Alayrac et al. [2022] showed that additionally training them on interleaved
sequences of text and images can lead to the emergence of in-context learning
capabilities. However, the dataset they used, M3W, is not public and is only in
English. There have been attempts to reproduce their results but the released
datasets are English-only. In contrast, current multilingual and multimodal
datasets are either composed of caption-like only or medium-scale or fully
private data. This limits mLLM research for the 7,000 other languages spoken in
the world. We therefore introduce mOSCAR, to the best of our knowledge the
first large-scale multilingual and multimodal document corpus crawled from the
web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
is sufficiently safe, diverse and of good quality. We additionally train two
types of multilingual model to prove the benefits of mOSCAR: (1) a model
trained on a subset of mOSCAR and captioning data and (2) a model train on
captioning data only. The model additionally trained on mOSCAR shows a strong
boost in few-shot learning performance across various multilingual image-text
tasks and benchmarks, confirming previous findings for English-only mLLMs.

Multimodal Large Language Models (mLLMs) that are trained on caption-like and interleaved text-image data, such as mOSCAR, show improved in-context learning capabilities, boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, and address the limitation of current multilingual and multimodal datasets.