Grounding language to the visual observations of a navigating agent can be
performed using off-the-shelf visual-language models pretrained on
Internet-scale data (e.g., image captions). While this is useful for matching
images to natural language descriptions of object goals, it remains disjoint
from the process of mapping the environment, so that it lacks the spatial
precision of classic geometric maps. To address this problem, we propose
VLMaps, a spatial map representation that directly fuses pretrained
visual-language features with a 3D reconstruction of the physical world. VLMaps
can be autonomously built from video feed on robots using standard exploration
approaches and enables natural language indexing of the map without additional
labeled data. Specifically, when combined with large language models (LLMs),
VLMaps can be used to (i) translate natural language commands into a sequence
of open-vocabulary navigation goals (which, beyond prior work, can be spatial
by construction, e.g., "in between the sofa and TV" or "three meters to the
right of the chair") directly localized in the map, and (ii) can be shared
among multiple robots with different embodiments to generate new obstacle maps
on-the-fly (by using a list of obstacle categories). Extensive experiments
carried out in simulated and real world environments show that VLMaps enable
navigation according to more complex language instructions than existing
methods. Videos are available at this https URL.

本研究提出了 VLMaps，一种将 3D 物理世界重建与预训练的视觉语言特征直接融合的空间地图表示，其能够通过大型语言模型将自然语言命令直接转化为空间导航目标。VLMaps 能够在不需要额外标注数据的情况下自主构建，并且可以与多种类型的机器人共享，能够进行导航的同时生成新的障碍物地图。实验表明，VLMaps 可以支持人类语言含义更丰富的导航指令。