In recent years, vision and language pre-training (VLP) models have advanced
the state-of-the-art results in a variety of cross-modal downstream tasks.
Aligning cross-modal semantics is claimed to be one of the essential
capabilities of VLP models. However, it still remains unclear about the inner
working mechanism of alignment in VLP models. In this paper, we propose a new
probing method that is based on image captioning to first empirically study the
cross-modal semantics alignment of VLP models. Our probing method is built upon
the fact that given an image-caption pair, the VLP models will give a score,
indicating how well two modalities are aligned; maximizing such scores will
generate sentences that VLP models believe are of good alignment. Analyzing
these sentences thus will reveal in what way different modalities are aligned
and how well these alignments are in VLP models. We apply our probing method to
five popular VLP models, including UNITER, ROSITA, ViLBERT, CLIP, and LXMERT,
and provide a comprehensive analysis of the generated captions guided by these
models. Our results show that VLP models (1) focus more on just aligning
objects with visual words, while neglecting global semantics; (2) prefer fixed
sentence patterns, thus ignoring more important textual information including
fluency and grammar; and (3) deem the captions with more visual words are
better aligned with images. These findings indicate that VLP models still have
weaknesses in cross-modal semantics alignment and we hope this work will draw
researchers' attention to such problems when designing a new VLP model.

本文提出了一种基于图像字幕生成的新型探测方法，用于研究视觉语言预训练模型中跨模态语义对齐的内部机制，发现 VLP 模型对齐的主要是对象和视觉词，忽略了全局语义，还存在固定的句子模式，无视语法和流畅性等问题。