Web-crawled datasets are pivotal to the success of pre-training
vision-language models, exemplified by CLIP. However, web-crawled AltTexts can
be noisy and potentially irrelevant to images, thereby undermining the crucial
image-text alignment. Existing methods for rewriting captions using large
language models (LLMs) have shown promise on small, curated datasets like CC3M
and CC12M. Nevertheless, their efficacy on massive web-captured captions is
constrained by the inherent noise and randomness in such data. In this study,
we address this limitation by focusing on two key aspects: data quality and
data variety. Unlike recent LLM rewriting techniques, we emphasize exploiting
visual concepts and their integration into the captions to improve data
quality. For data variety, we propose a novel mixed training scheme that
optimally leverages AltTexts alongside newly generated Visual-enriched Captions
(VeC). We use CLIP as one example and adapt the method for CLIP training on
large-scale web-crawled datasets, named VeCLIP. We conduct a comprehensive
evaluation of VeCLIP across small, medium, and large scales of raw data. Our
results show significant advantages in image-text alignment and overall model
performance, underscoring the effectiveness of VeCLIP in improving CLIP
training. For example, VeCLIP achieves a remarkable over 20% improvement in
COCO and Flickr30k retrieval tasks under the 12M setting. For data efficiency,
we also achieve a notable over 3% improvement while using only 14% of the data
employed in the vanilla CLIP and 11% in ALIGN.

本研究关注于通过改善数据质量和数据多样性，特别强调了视觉概念与标题的整合，提出了一种用于 web 爬取数据集训练的新方法 VeCLIP，通过综合评估数据效率和模型性能，证明了 VeCLIP 在改善图片 - 文本对齐和整体模型性能方面的显著优势。

从稀缺到高效：通过视觉丰富的标题改进 CLIP 训练

From Scarcity to Efficiency: Improving CLIP Training via Visual-enriched  Captions

Web-crawled datasets have enabled remarkable generalization capabilities in
recent image-text models such as CLIP (Contrastive Language-Image pre-training)
or Flamingo, but little is known about the dataset creation processes. In this
work, we introduce a testbed of six publicly available data sources - YFCC,
LAION, Conceptual Captions, WIT, RedCaps, Shutterstock - to investigate how
pre-training distributions induce robustness in CLIP. We find that the
performance of the pre-training data varies substantially across distribution
shifts, with no single data source dominating. Moreover, we systematically
study the interactions between these data sources and find that combining
multiple sources does not necessarily yield better models, but rather dilutes
the robustness of the best individual data source. We complement our empirical
findings with theoretical insights from a simple setting, where combining the
training data also results in diluted robustness. In addition, our theoretical
model provides a candidate explanation for the success of the CLIP-based data
filtering technique recently employed in the LAION dataset. Overall our results
demonstrate that simply gathering a large amount of data from the web is not
the most effective way to build a pre-training dataset for robust
generalization, necessitating further study into dataset design. Code is
available at this https URL

探究六种公开数据来源 ——YFCC、LAION、Conceptual Captions、WIT、RedCaps、Shutterstock—— 对 CLIP 进行预训练的分布移位是否会对性能产生影响，发现不同的预训练数据表现相差很大，并且多个数据源的组合并不一定产生更好的模型，提出从理论和实践两方面着手进行数据集设计以实现稳健一般化训练的需求。