Large training sets have become a cornerstone of machine learning and are the
foundation for recent advances in language modeling and multimodal learning.
While data curation for pre-training is often still ad-hoc, one common paradigm
is to first collect a massive pool of data from the Web and then filter this
candidate pool down to an actual training set via various heuristics. In this
work, we study the problem of learning a data filtering network (DFN) for this
second step of filtering a large uncurated dataset. Our key finding is that the
quality of a network for filtering is distinct from its performance on
downstream tasks: for instance, a model that performs well on ImageNet can
yield worse training sets than a model with low ImageNet accuracy that is
trained on a small amount of high-quality data. Based on our insights, we
construct new data filtering networks that induce state-of-the-art image-text
datasets. Specifically, our best performing dataset DFN-5B enables us to train
state-of-the-art models for their compute budgets: among other improvements on
a variety of tasks, a ViT-H trained on our dataset achieves 83.0% zero-shot
transfer accuracy on ImageNet, out-performing models trained on other datasets
such as LAION-2B, DataComp-1B, or OpenAI's WIT. In order to facilitate further
research in dataset design, we also release a new 2 billion example dataset
DFN-2B and show that high performance data filtering networks can be trained
from scratch using only publicly available data.

通过构建数据过滤网络，该论文研究了大训练集上数据筛选的问题，并基于该网络构建了新的图像 - 文本数据集，为状态 - of-the-art 模型训练提供了高性能数据集，同时还释放了可从公开数据从头训练高性能数据过滤网络的新的 200 亿样本数据集，以促进数据集设计的进一步研究。