In recent years, data selection has emerged as a core issue for large-scale
visual-language model pretraining, especially on noisy web-curated datasets.
One widely adopted strategy assigns quality scores such as CLIP similarity for
each sample and retains the data pairs with the highest scores. However, these
approaches are agnostic of data distribution and always fail to select the most
informative samples. To solve this problem, we propose a simple yet
theoretically principled metric named Variance Alignment Score (VAS), which has
the form $\langle \Sigma_{\text{test}}, \Sigma_i\rangle$. Here,
$\Sigma_{\text{test}}$ represents the target (cross-)covariance matrix we aim
to align, potentially based on prior knowledge, while $\Sigma_i$ denotes the
tensor product of single or multi-modal representations for the $i$-th sample.
We further design a new data selection method that maximizes the total VAS. We
provide theoretical analysis in a simplified setting to demonstrate the
theoretical advantage of VAS over random or other existing data selection.
Experimentally, applying VAS and CLIP scores together can outperform baselines
by a margin of $1.3\%$ average on 38 evaluation sets for noisy dataset DataComp
and $2.5\%$ on VTAB for high-quality dataset CC12M. Additionally, our ablation
study also shows visual features are better than text for calculating VAS, and
the related classical experimental design methods may fail under this context.

通过提出一种简单而理论上有根据的指标 Variance Alignment Score (VAS)，本研究解决了大规模视觉语言模型预训练中数据选择的核心问题，并设计了一种最大化 VAS 的新数据选择方法，实验证明 VAS 和 CLIP 分数的联合应用在嘈杂数据集 DataComp 上的 38 个评估集中平均提升了 1.3％，在高质量数据集 CC12M 上提升了 2.5％，此外，本研究还表明视觉特征在计算 VAS 时优于文本，并且相关的经典实验设计方法在这种情况下可能失败。

方差对齐得分：一种简单但难以超越的多模态对比学习数据选择方法

Variance Alignment Score: A Simple But Tough-to-Beat Data Selection  Method for Multimodal Contrastive Learning

As sharing images in an instant message is a crucial factor, there has been
active research on learning a image-text multi-modal dialogue model. However,
training a well-generalized multi-modal dialogue model is challenging because
existing multi-modal dialogue datasets contain a small number of data, limited
topics, and a restricted variety of images per dialogue. In this paper, we
present a multi-modal dialogue dataset creation pipeline that involves matching
large-scale images to dialogues based on CLIP similarity. Using this automatic
pipeline, we propose a large-scale multi-modal dialogue dataset, DialogCC,
which covers diverse real-world topics and various images per dialogue. With
extensive experiments, we demonstrate that training a multi-modal dialogue
model with our dataset can improve generalization performance. Additionally,
existing models trained with our dataset achieve state-of-the-art performance
on image and text retrieval tasks. The source code and the dataset will be
released after publication.

本文介绍了一种基于 CLIP 相似度的多模态对话数据集创建管道，使用这个管道，我们提出了一个大规模的多模态对话数据集 DialogCC，并且通过广泛的实验结果表明，使用我们的数据集训练多模态对话模型可以改善泛化性能，与此同时，使用我们的数据集训练的现有模型在图像和文本检索任务上取得了最先进的表现。