Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pretraining for two recent VL models using a suite of analyses (clustering, probing, and performance on a commonsense question answering task) in a language-only setting. We find that the multimodal models fail to significantly outperform the text-only variants, suggesting that future work is required if multimodal pretraining is to be pursued as a means of improving NLP in general.

本文研究比较基于视觉和语言的预训练模型和仅基于文本的预训练模型的语义表示，结果发现基于视觉和语言的模型在仅语言方面无法显著优于仅基于文本的模型，因此这种多模态预训练对于提高自然语言处理的效果仍需要进一步研究。

视觉语言预训练是否提高了词汇连接能力?