Language-image pre-training largely relies on how precisely and thoroughly a
text describes its paired image. In practice, however, the contents of an image
can be so rich that well describing them requires lengthy captions (e.g., with
10 sentences), which are usually missing in existing datasets. Consequently,
there are currently no clear evidences on whether and how language-image
pre-training could benefit from long captions. To figure this out, we first
re-caption 30M images with detailed descriptions using a pre-trained
Multi-modality Large Language Model (MLLM), and then study the usage of the
resulting captions under a contrastive learning framework. We observe that,
each sentence within a long caption is very likely to describe the image
partially (e.g., an object). Motivated by this, we propose to dynamically
sample sub-captions from the text label to construct multiple positive pairs,
and introduce a grouping loss to match the embeddings of each sub-caption with
its corresponding local image patches in a self-supervised manner. Experimental
results on a wide rage of downstream tasks demonstrate the consistent
superiority of our method, termed DreamLIP, over previous alternatives,
highlighting its fine-grained representational capacity. It is noteworthy that,
on the tasks of image-text retrieval and semantic segmentation, our model
trained with 30M image-text pairs achieves on par or even better performance
than CLIP trained with 400M pairs. Project page is available at
this https URL

通过在大规模语言模型中生成详细描述的长标题，我们提出了一种动态采样子标题的方法，以在对比学习框架中构建多个正向对并使用分组损失进行自监督训练，实验证明该方法在多种下游任务上具有明显优势。