Large training sets have become a cornerstone of machine learning and are the
foundation for recent advances in language modeling and multimodal learning.
While data curation for pre-training is often still ad-hoc, one common paradigm
is to first collect a massive pool of data from the Web and then filter this
candidate pool down to an actual training set via various heuristics. In this
work, we study the problem of learning a data filtering network (DFN) for this
second step of filtering a large uncurated dataset. Our key finding is that the
quality of a network for filtering is distinct from its performance on
downstream tasks: for instance, a model that performs well on ImageNet can
yield worse training sets than a model with low ImageNet accuracy that is
trained on a small amount of high-quality data. Based on our insights, we
construct new data filtering networks that induce state-of-the-art image-text
datasets. Specifically, our best performing dataset DFN-5B enables us to train
state-of-the-art models for their compute budgets: among other improvements on
a variety of tasks, a ViT-H trained on our dataset achieves 83.0% zero-shot
transfer accuracy on ImageNet, out-performing models trained on other datasets
such as LAION-2B, DataComp-1B, or OpenAI's WIT. In order to facilitate further
research in dataset design, we also release a new 2 billion example dataset
DFN-2B and show that high performance data filtering networks can be trained
from scratch using only publicly available data.

通过构建数据过滤网络，该论文研究了大训练集上数据筛选的问题，并基于该网络构建了新的图像 - 文本数据集，为状态 - of-the-art 模型训练提供了高性能数据集，同时还释放了可从公开数据从头训练高性能数据过滤网络的新的 200 亿样本数据集，以促进数据集设计的进一步研究。

数据过滤网络

Data Filtering Networks

While a number of different algorithms have recently been proposed for
convolutional dictionary learning, this remains an expensive problem. The
single biggest impediment to learning from large training sets is the memory
requirements, which grow at least linearly with the size of the training set
since all existing methods are batch algorithms. The work reported here
addresses this limitation by extending online dictionary learning ideas to the
convolutional context.

本文研究了针对卷积字典学习的在线学习方法，解决了传统批量算法在处理大规模训练数据时所需的较高内存要求的问题。