Data selection has emerged as a core issue for large-scale visual-language
model pretaining (e.g., CLIP), particularly with noisy web-curated datasets.
Three main data selection approaches are: (1) leveraging external non-CLIP
models to aid data selection, (2) training new CLIP-style embedding models that
are more effective at selecting high-quality data than the original OpenAI CLIP
model, and (3) designing better metrics or strategies universally applicable to
any CLIP embedding without requiring specific model properties (e.g., CLIPScore
is one popular metric). While the first two approaches have been extensively
studied, the third remains under-explored. In this paper, we advance the third
approach by proposing two new methods. Firstly, instead of classical CLIP
scores that only consider the alignment between two modalities from a single
sample, we introduce negCLIPLoss, a CLIP loss-inspired method that adds the
alignment between one sample and its contrastive pairs as an extra
normalization term for better quality measurement. Secondly, when downstream
tasks are known, we propose a new norm-based metric, NormSim, to measure the
similarity between pretraining data and target data. We test our methods on the
data selection benchmark, DataComp~\cite{gadre2023datacomp}. Compared to the
best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\%
improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation
tasks. Moreover, both negCLIPLoss and NormSim are compatible with existing
techniques. By combining our methods with the current best methods
DFN~\cite{fang2023data} and HYPE~\cite{kim2024hype}, we can boost average
performance on downstream tasks by 0.9\%, achieving a new state-of-the-art.

数据选择是大规模视觉 - 语言模型预训练（例如 CLIP）中的一个核心问题，本文提出了两种新方法：一种是利用 negCLIPLoss 作为质量测量的 CLIP 损失启发式方法，另一种是使用 NormSim 度量预训练数据和目标数据之间的相似性，这两种方法的结合将当前最佳方法 DFN 和 HYPE 的平均性能提高了 0.9％，达到了新的最佳水平。