Image-text interleaved data, consisting of multiple images and texts arranged
in a natural document format, aligns with the presentation paradigm of internet
data and closely resembles human reading habits. Recent studies have shown that
such data aids multimodal in-context learning and maintains the capabilities of
large language models during multimodal fine-tuning. However, the limited scale
and diversity of current image-text interleaved data restrict the development
of multimodal large language models. In this paper, we introduce OmniCorpus, a
10 billion-scale image-text interleaved dataset. Using an efficient data
engine, we filter and extract large-scale high-quality documents, which contain
8.6 billion images and 1,696 billion text tokens. Compared to counterparts
(e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while
maintaining good data quality; 2) features more diverse sources, including both
English and non-English websites as well as video-centric websites; 3) is more
flexible, easily degradable from an image-text interleaved format to pure text
corpus and image-text pairs. Through comprehensive analysis and experiments, we
validate the quality, usability, and effectiveness of the proposed dataset. We
hope this could provide a solid data foundation for future multimodal model
research. Code and data are released at
this https URL

介绍了 OmniCorpus 数据集，一个 100 亿级别的图文交错数据集，与互联网数据的呈现范式相一致并且紧密符合人类阅读习惯，通过全面的分析和实验验证了所提出数据集的质量、可用性和有效性，为未来的多模态模型研究提供了坚实的数据基础。