This paper shows that text-only Language Models (LM) can learn to ground
spatial relations like "left of" or "below" if they are provided with explicit
location information of objects and they are properly trained to leverage those
locations. We perform experiments on a verbalized version of the Visual Spatial
Reasoning (VSR) dataset, where images are coupled with textual statements which
contain real or fake spatial relations between two objects of the image. We
verbalize the images using an off-the-shelf object detector, adding location
tokens to every object label to represent their bounding boxes in textual form.
Given the small size of VSR, we do not observe any improvement when using
locations, but pretraining the LM over a synthetic dataset automatically
derived by us improves results significantly when using location tokens. We
thus show that locations allow LMs to ground spatial relations, with our
text-only LMs outperforming Vision-and-Language Models and setting the new
state-of-the-art for the VSR dataset. Our analysis show that our text-only LMs
can generalize beyond the relations seen in the synthetic dataset to some
extent, learning also more useful information than that encoded in the spatial
rules we used to create the synthetic dataset itself.

文中通过提供对象的显式位置信息并进行适当训练，展示出仅文本的语言模型（Language Models）可以学习到类似 “左侧” 或 “下方” 等空间关系。通过对视觉空间推理（Visual Spatial Reasoning）数据集的一个口语化版本进行实验，其中图像与包含真实或虚假空间关系的文本语句相配对。作者使用现有的物体检测器给图像增添了位置标记，以文本形式表示每个物体的边界框。尽管视觉空间推理数据集较小，但使用位置信息并无明显改进，然而，通过在由作者衍生的合成数据集上进行预训练，使用位置标记的结果显著改善。作者因此展示了位置信息使语言模型能够对空间关系进行基础性的理解，仅文本的语言模型表现优于视觉与语言模型，成为视觉空间推理数据集的最新技术成果。作者的分析显示，仅文本的语言模型在一定程度上能够推广到合成数据集中未出现的关系，并且还学习了比我们用来创建合成数据集的空间规则中所编码的更有用的信息。

基于纯文本语言模型的空间关系基础

Grounding Spatial Relations in Text-Only Language Models

Spatial relations are a basic part of human cognition. However, they are
expressed in natural language in a variety of ways, and previous work has
suggested that current vision-and-language models (VLMs) struggle to capture
relational information. In this paper, we present Visual Spatial Reasoning
(VSR), a dataset containing more than 10k natural text-image pairs with 66
types of spatial relations in English (such as: under, in front of, and
facing). While using a seemingly simple annotation format, we show how the
dataset includes challenging linguistic phenomena, such as varying reference
frames. We demonstrate a large gap between human and model performance: the
human ceiling is above 95%, while state-of-the-art models only achieve around
70%. We observe that VLMs' by-relation performances have little correlation
with the number of training examples and the tested models are in general
incapable of recognising relations concerning the orientations of objects.

本研究提出 Visual Spatial Reasoning（VSR）数据集，这是包含超过 10k 已标注的英文自然文本图像对和 66 种空间关系的数据集，研究表明当前视觉语言模型只能达到约 70％的准确率，无法识别有关物体朝向的关系。