Web-crawled datasets have enabled remarkable generalization capabilities in
recent image-text models such as CLIP (Contrastive Language-Image pre-training)
or Flamingo, but little is known about the dataset creation processes. In this
work, we introduce a testbed of six publicly available data sources - YFCC,
LAION, Conceptual Captions, WIT, RedCaps, Shutterstock - to investigate how
pre-training distributions induce robustness in CLIP. We find that the
performance of the pre-training data varies substantially across distribution
shifts, with no single data source dominating. Moreover, we systematically
study the interactions between these data sources and find that combining
multiple sources does not necessarily yield better models, but rather dilutes
the robustness of the best individual data source. We complement our empirical
findings with theoretical insights from a simple setting, where combining the
training data also results in diluted robustness. In addition, our theoretical
model provides a candidate explanation for the success of the CLIP-based data
filtering technique recently employed in the LAION dataset. Overall our results
demonstrate that simply gathering a large amount of data from the web is not
the most effective way to build a pre-training dataset for robust
generalization, necessitating further study into dataset design. Code is
available at this https URL

探究六种公开数据来源 ——YFCC、LAION、Conceptual Captions、WIT、RedCaps、Shutterstock—— 对 CLIP 进行预训练的分布移位是否会对性能产生影响，发现不同的预训练数据表现相差很大，并且多个数据源的组合并不一定产生更好的模型，提出从理论和实践两方面着手进行数据集设计以实现稳健一般化训练的需求。