Recent visuolinguistic pre-trained models show promising progress on various
end tasks such as image retrieval and video captioning. Yet, they fail
miserably on the recently proposed Winoground dataset, which challenges models
to match paired images and English captions, with items constructed to overlap
lexically but differ in meaning (e.g., "there is a mug in some grass" vs.
"there is some grass in a mug"). By annotating the dataset using new
fine-grained tags, we show that solving the Winoground task requires not just
compositional language understanding, but a host of other abilities like
commonsense reasoning or locating small, out-of-focus objects in low-resolution
images. In this paper, we identify the dataset's main challenges through a
suite of experiments on related tasks (probing task, image retrieval task),
data augmentation, and manual inspection of the dataset. Our analysis suggests
that a main challenge in visuolinguistic models may lie in fusing visual and
textual representations, rather than in compositional language understanding.
We release our annotation and code at
this https URL .

通过一系列实验和数据分析，发现视觉和语言模型中融合文字和视觉表示的挑战可能不在于组合性语言理解，而在于其他能力，如常识推理、低分辨率图像中小物体的定位等，并提出数据扩充等解决方案。