Vision Language Models (VLMs) have achieved impressive performance in 2D
image understanding, however they are still struggling with spatial
understanding which is the foundation of Embodied AI. In this paper, we propose
SpatialBot for better spatial understanding by feeding both RGB and depth
images. Additionally, we have constructed the SpatialQA dataset, which involves
multi-level depth-related questions to train VLMs for depth understanding.
Finally, we present SpatialBench to comprehensively evaluate VLMs' capabilities
in spatial understanding at different levels. Extensive experiments on our
spatial-understanding benchmark, general VLM benchmarks and Embodied AI tasks,
demonstrate the remarkable improvements of SpatialBot trained on SpatialQA. The
model, code and data are available at this https URL

通过为 Vision Language Models 提供 RGB 和深度图像，我们提出了 SpatialBot 来改进其对空间的理解能力，并构建了 SpatialQA 数据集用于训练深度理解，最后通过 SpatialBench 对其在不同层次上的空间理解能力进行综合评估，实验结果表明在 SpatialQA 数据集上训练的 SpatialBot 在空间理解方面取得了显著的改进。