We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple
method for pretraining more fine-grained multimodal representations from
image-text pairs. Given that multiple image patches often correspond to single
words, we propose to learn a grouping of image patches for every token in the
caption. To achieve this, we use a sparse similarity metric between image
patches and language tokens and compute for each token a language-grouped
vision embedding as the weighted average of patches. The token and
language-grouped vision embeddings are then contrasted through a fine-grained
sequence-wise loss that only depends on individual samples and does not require
other batch samples as negatives. This enables more detailed information to be
learned in a computationally inexpensive manner. SPARC combines this
fine-grained loss with a contrastive loss between global image and text
embeddings to learn representations that simultaneously encode global and local
information. We thoroughly evaluate our proposed method and show improved
performance over competing approaches both on image-level tasks relying on
coarse-grained information, e.g. classification, as well as region-level tasks
relying on fine-grained information, e.g. retrieval, object detection, and
segmentation. Moreover, SPARC improves model faithfulness and captioning in
foundational vision-language models.

我们介绍 SPARC (SPARse Fine-grained Contrastive Alignment)，这是一种从图像 - 文本对中预训练更细粒度多模态表示的简单方法。SPARC 结合了精细化的序列损失和对全局图像和文本嵌入进行对比的损失，以同时编码全局和局部信息。我们对这种方法进行了全面评估，并显示在依赖于粗粒度信息的图像级任务（如分类）以及依赖于细粒度信息的区域级任务（如检索、目标检测和分割）上改善了性能。此外，SPARC 改善了模型的准确性和生成图像描述的能力。