The massive growth of image-text data through web crawling inherently
presents the challenge of variability in data quality. This paper introduces a
novel algorithm, rooted in human knowledge, to compress this vast corpus of
web-crawled image-text datasets to a compact and high-quality form. Our method
unfolds in three major steps. First, we collect an image-text dataset, wherein
each image is associated with multiple captions sourced from diverse origins.
Then, to systemically capture human preferences regarding the best caption
paired with each image, we establish a comprehensive set of both subjective and
objective criteria for critically guiding the alignment assessment from
labelers. Lastly, we train a reward model on the annotated dataset to
internalize the nuanced human understanding of image-text alignment. The
resulting reward model thus can act as a human-like referee to filter
misaligned/low-quality image-text pairs. Extensive experiments demonstrate that
we are able to secure (or even improve) model performance by compressing the
image-text datasets up to ~90%. An impressive example is that, by aggressively
reducing the total training sample from 130M to 15.5M (e.g., ~9x smaller), our
BLIP-B/16 models still consistently show superior performance compared with the
full-size-dataset counterpart on image-text retrieval (Flickr30K, COCO) by
~2.5% in Recall@1, and on image-captioning (Nocaps, COCO) by ~10.0% in CIDEr
and ~2.7% in SPICE.

该研究通过采用人工智能算法对图像文本数据进行高质量压缩，并利用训练出的奖励模型作为人类般的裁判来过滤不对齐 / 低质量的图像文本对。

压缩与对齐：用人类知识筛选图像文本数据

Compress & Align: Curating Image-Text Data with Human Knowledge

This paper evaluates large georeferenced corpora, taken from both web-crawled
and social media sources, against ground-truth population and language-census
datasets. The goal is to determine (i) which dataset best represents population
demographics; (ii) in what parts of the world the datasets are most
representative of actual populations; and (iii) how to weight the datasets to
provide more accurate representations of underlying populations. The paper
finds that the two datasets represent very different populations and that they
correlate with actual populations with values of r=0.60 (social media) and
r=0.49 (web-crawled). Further, Twitter data makes better predictions about the
inventory of languages used in each country.

本文通过评估地理信息标注的大型语料库，探究不同数据集在代表人口统计学中的差异性和效果，并发现社交媒体数据在预测语言库存方面比网络数据更准确。