The success of large-scale contextual language models has attracted great
interest in probing what is encoded in their representations. In this work, we
consider a new question: to what extent contextual representations of concrete
nouns are aligned with corresponding visual representations? We design a
probing model that evaluates how effective are text-only representations in
distinguishing between matching and non-matching visual representations. Our
findings show that language representations alone provide a strong signal for
retrieving image patches from the correct object categories. Moreover, they are
effective in retrieving specific instances of image patches; textual context
plays an important role in this process. Visually grounded language models
slightly outperform text-only language models in instance retrieval, but
greatly under-perform humans. We hope our analyses inspire future research in
understanding and improving the visual capabilities of language models.

本文通过设计一种探针模型，研究了上下文语言模型对应的具体名词在视觉表示方面的关联程度，并发现语言表示本身就具有检索正确对象类别的强信号以及检索图像特定实例的效果，而文本上下文在该过程中发挥了重要作用，但比起人类，具有图像为基础的语言模型在实例检索方面表现稍逊。我们希望这些结果能够启发未来的研究，进一步地理解和改善语言模型的视觉能力。