Despite the superior performance brought by vision-and-language pretraining,
it remains unclear whether learning with multi-modal data can help understand
each individual modality. In this work, we investigate how language can help
with visual representation learning from a probing perspective. Specifically,
we compare vision-and-language and vision-only models by probing their visual
representations on a broad range of tasks, in order to assess the quality of
the learned representations in a fine-grained manner. Interestingly, our
probing results suggest that vision-and-language models are better at label
prediction tasks like object and attribute prediction, while vision-only models
are stronger at dense prediction tasks that require more localized information.
With further analysis using detailed metrics, our study suggests that language
helps vision models learn better semantics, but not localization. Code is
released at this https URL.

我们通过探测的方式比较了视觉 - 语言模型和仅视觉模型的视觉表征，在广泛的任务范畴内评估学习表征的质量，发现视觉 - 语言模型更适合标签预测任务，而仅视觉模型更适合需要更多局部信息的密集预测任务，同时指出语言有利于视觉模型更好地学习语义，但不利于定位。