Large multimodal datasets have been instrumental in recent breakthroughs such
as CLIP, Stable Diffusion, and GPT-4. At the same time, datasets rarely receive
the same research attention as model architectures or training algorithms. To
address this shortcoming in the machine learning ecosystem, we introduce
DataComp, a benchmark where the training code is fixed and researchers innovate
by proposing new training sets. We provide a testbed for dataset experiments
centered around a new candidate pool of 12.8B image-text pairs from Common
Crawl. Participants in our benchmark design new filtering techniques or curate
new data sources and then evaluate their new dataset by running our
standardized CLIP training code and testing on 38 downstream test sets. Our
benchmark consists of multiple scales, with four candidate pool sizes and
associated compute budgets ranging from 12.8M to 12.8B samples seen during
training. This multi-scale design facilitates the study of scaling trends and
makes the benchmark accessible to researchers with varying resources.
Our baseline experiments show that the DataComp workflow is a promising way
of improving multimodal datasets. We introduce DataComp-1B, a dataset created
by applying a simple filtering algorithm to the 12.8B candidate pool. The
resulting 1.4B subset enables training a CLIP ViT-L/14 from scratch to 79.2%
zero-shot accuracy on ImageNet. Our new ViT-L/14 model outperforms a larger
ViT-g/14 trained on LAION-2B by 0.7 percentage points while requiring 9x less
training compute. We also outperform OpenAI's CLIP ViT-L/14 by 3.7 percentage
points, which is trained with the same compute budget as our model. These gains
highlight the potential for improving model performance by carefully curating
training sets. We view DataComp-1B as only the first step and hope that
DataComp paves the way toward the next generation of multimodal datasets.

DataComp 基准测试是一个围绕 Common Crawl 的约 128 亿个图像文本对的数据集实验平台，通过提出新的训练集来激励数据集的研究。通过应用简单的过滤算法，DataComp-1B 被创建并用于训练一个 CLIP ViT-L / 14 模型，使其在 ImageNet 上达到 79.2％的零 - shot 准确率，表明通过精心策划的训练集可能会提高模型的性能。