Spatial reasoning plays a vital role in both human cognition and machine
intelligence, prompting new research into language models' (LMs) capabilities
in this regard. However, existing benchmarks reveal shortcomings in evaluating
qualitative spatial reasoning (QSR). These benchmarks typically present
oversimplified scenarios or unclear natural language descriptions, hindering
effective evaluation. We present a novel benchmark for assessing QSR in LMs,
which is grounded in realistic 3D simulation data, offering a series of diverse
room layouts with various objects and their spatial relationships. This
approach provides a more detailed and context-rich narrative for spatial
reasoning evaluation, diverging from traditional, toy-task-oriented scenarios.
Our benchmark encompasses a broad spectrum of qualitative spatial
relationships, including topological, directional, and distance relations.
These are presented with different viewing points, varied granularities, and
density of relation constraints to mimic real-world complexities. A key
contribution is our logic-based consistency-checking tool, which enables the
assessment of multiple plausible solutions, aligning with real-world scenarios
where spatial relationships are often open to interpretation. Our benchmark
evaluation of advanced LMs reveals their strengths and limitations in spatial
reasoning. They face difficulties with multi-hop spatial reasoning and
interpreting a mix of different view descriptions, pointing to areas for future
improvement.

我们提出了一个新的基准评估语言模型中的定性空间推理，该基准建立在逼真的三维仿真数据上，提供了一系列具有多样化房间布局、不同对象及其空间关系的详细和富有背景的叙述。我们的基准评估了先进的语言模型在空间推理方面的优势和局限性，发现它们在多跳空间推理和解释混合视角描述方面存在困难，并指出了未来改进的方向。