The availability of large-scale image captioning and visual question
answering datasets has contributed significantly to recent successes in
vision-and-language pre-training. However, these datasets are often collected
with overrestrictive requirements inherited from their original target tasks
(e.g., image caption generation), which limit the resulting dataset scale and
diversity. We take a step further in pushing the limits of vision-and-language
pre-training data by relaxing the data collection pipeline used in Conceptual
Captions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M
(CC12M), a dataset with 12 million image-text pairs specifically meant to be
used for vision-and-language pre-training. We perform an analysis of this
dataset and benchmark its effectiveness against CC3M on multiple downstream
tasks with an emphasis on long-tail visual recognition. Our results clearly
illustrate the benefit of scaling up pre-training data for vision-and-language
tasks, as indicated by the new state-of-the-art results on both the nocaps and
Conceptual Captions benchmarks.

通过松弛 Conceptual Captions 3M (CC3M) [Sharma et al. 2018] 数据收集流程，我们引入了 Conceptual 12M（CC12M）数据集，并通过针对长尾视觉识别的多个下游任务基准测试其有效性，结果表明增加预训练数据规模会使视觉和语言任务更加有效。