Data selection has emerged as a core issue for large-scale visual-language
model pretaining (e.g., CLIP), particularly with noisy web-curated datasets.
Three main data selection approaches are: (1) leveraging external non-CLIP
models to aid data selection, (2) training new CLIP-style embedding models that
are more effective at selecting high-quality data than the original OpenAI CLIP
model, and (3) designing better metrics or strategies universally applicable to
any CLIP embedding without requiring specific model properties (e.g., CLIPScore
is one popular metric). While the first two approaches have been extensively
studied, the third remains under-explored. In this paper, we advance the third
approach by proposing two new methods. Firstly, instead of classical CLIP
scores that only consider the alignment between two modalities from a single
sample, we introduce negCLIPLoss, a CLIP loss-inspired method that adds the
alignment between one sample and its contrastive pairs as an extra
normalization term for better quality measurement. Secondly, when downstream
tasks are known, we propose a new norm-based metric, NormSim, to measure the
similarity between pretraining data and target data. We test our methods on the
data selection benchmark, DataComp~\cite{gadre2023datacomp}. Compared to the
best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\%
improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation
tasks. Moreover, both negCLIPLoss and NormSim are compatible with existing
techniques. By combining our methods with the current best methods
DFN~\cite{fang2023data} and HYPE~\cite{kim2024hype}, we can boost average
performance on downstream tasks by 0.9\%, achieving a new state-of-the-art.

数据选择是大规模视觉 - 语言模型预训练（例如 CLIP）中的一个核心问题，本文提出了两种新方法：一种是利用 negCLIPLoss 作为质量测量的 CLIP 损失启发式方法，另一种是使用 NormSim 度量预训练数据和目标数据之间的相似性，这两种方法的结合将当前最佳方法 DFN 和 HYPE 的平均性能提高了 0.9％，达到了新的最佳水平。

多模态对比学习的 CLIPLoss 和基于范数的数据选择方法

CLIPLoss and Norm-Based Data Selection Methods for Multimodal  Contrastive Learning

The quality of pre-training data plays a critical role in the performance of
foundation models. Popular foundation models often design their own recipe for
data filtering, which makes it hard to analyze and compare different data
filtering approaches. DataComp is a new benchmark dedicated to evaluating
different methods for data filtering. This paper describes our learning and
solution when participating in the DataComp challenge. Our filtering strategy
includes three stages: single-modality filtering, cross-modality filtering, and
data distribution alignment. We integrate existing methods and propose new
solutions, such as computing CLIP score on horizontally flipped images to
mitigate the interference of scene text, using vision and language models to
retrieve training samples for target downstream tasks, rebalancing the data
distribution to improve the efficiency of allocating the computational budget,
etc. We slice and dice our design choices, provide in-depth analysis, and
discuss open questions. Our approach outperforms the best method from the
DataComp paper by over 4% on the average performance of 38 tasks and by over 2%
on ImageNet.

本论文介绍了在参加 DataComp 挑战时的学习和解决方案，通过三个阶段的筛选策略与已有方法整合以及提出新的解决方案，如使用水平翻转图像计算 CLIP 分数以减轻场景文字的干扰、利用视觉和语言模型检索目标下游任务的训练样本，重新平衡数据分布以提高计算预算分配效率等，获得了比 DataComp 论文中最佳方法平均性能提高 4% 的成绩，以及在 ImageNet 上提高 2% 以上的成绩。