Spatial relations are a basic part of human cognition. However, they are
expressed in natural language in a variety of ways, and previous work has
suggested that current vision-and-language models (VLMs) struggle to capture
relational information. In this paper, we present Visual Spatial Reasoning
(VSR), a dataset containing more than 10k natural text-image pairs with 66
types of spatial relations in English (such as: under, in front of, and
facing). While using a seemingly simple annotation format, we show how the
dataset includes challenging linguistic phenomena, such as varying reference
frames. We demonstrate a large gap between human and model performance: the
human ceiling is above 95%, while state-of-the-art models only achieve around
70%. We observe that VLMs' by-relation performances have little correlation
with the number of training examples and the tested models are in general
incapable of recognising relations concerning the orientations of objects.

本研究提出 Visual Spatial Reasoning（VSR）数据集，这是包含超过 10k 已标注的英文自然文本图像对和 66 种空间关系的数据集，研究表明当前视觉语言模型只能达到约 70％的准确率，无法识别有关物体朝向的关系。