Existing image-text modality alignment in Vision Language Models (VLMs)
treats each text token equally in an autoregressive manner. Despite being
simple and effective, this method results in sub-optimal cross-modal alignment
by over-emphasizing the text tokens that are less correlated with or even
contradictory with the input images. In this paper, we advocate for assigning
distinct contributions for each text token based on its visual correlation.
Specifically, we present by contrasting image inputs, the difference in
prediction logits on each text token provides strong guidance of visual
correlation. We therefore introduce Contrastive ALignment (CAL), a simple yet
effective re-weighting strategy that prioritizes training visually correlated
tokens. Our experimental results demonstrate that CAL consistently improves
different types of VLMs across different resolutions and model sizes on various
benchmark datasets. Importantly, our method incurs minimal additional
computational overhead, rendering it highly efficient compared to alternative
data scaling strategies. Codes are available at
this https URL

通过对比图像输入，对每个文本标记的预测值差异提供了强大的视觉关联指导，我们提出了对每个文本标记赋予不同贡献的方法，名为 Contrastive ALignment (CAL)。我们的实验证明，CAL 方法在各种基准数据集上始终改善了不同类型的 VLM，并且与其他数据调整策略相比，我们的方法具有最小的额外计算开销。