Multimodal alignment between language and vision is the fundamental topic in
current vision-language model research. Contrastive Captioners (CoCa), as a
representative method, integrates Contrastive Language-Image Pretraining (CLIP)
and Image Caption (IC) into a unified framework, resulting in impressive
results. CLIP imposes a bidirectional constraints on global representation of
entire images and sentences. Although IC conducts an unidirectional
image-to-text generation on local representation, it lacks any constraint on
local text-to-image reconstruction, which limits the ability to understand
images at a fine-grained level when aligned with texts. To achieve multimodal
alignment from both global and local perspectives, this paper proposes
Symmetrizing Contrastive Captioners (SyCoCa), which introduces bidirectional
interactions on images and texts across the global and local representation
levels. Specifically, we expand a Text-Guided Masked Image Modeling (TG-MIM)
head based on ITC and IC heads. The improved SyCoCa can further leverage
textual cues to reconstruct contextual images and visual cues to predict
textual contents. When implementing bidirectional local interactions, the local
contents of images tend to be cluttered or unrelated to their textual
descriptions. Thus, we employ an attentive masking strategy to select effective
image patches for interaction. Extensive experiments on five vision-language
tasks, including image-text retrieval, image-captioning, visual question
answering, and zero-shot/finetuned image classification, validate the
effectiveness of our proposed method.

当前视觉语言模型研究的基础主题是语言和视觉之间的多模态对齐。对比式字幕生成器 (CoCa) 是一种代表性方法，它将对比语言 - 图像预训练 (CLIP) 和图像字幕 (IC) 集成到统一框架中，取得了令人印象深刻的结果。本文提出了一种称为对称式对比式字幕生成器 (SyCoCa) 的方法，从全局和局部表达水平上引入图像和文本交互，以实现多模态对齐。在实验中，我们验证了所提出方法的有效性。