For robots to perform a wide variety of tasks, they require a 3D
representation of the world that is semantically rich, yet compact and
efficient for task-driven perception and planning. Recent approaches have
attempted to leverage features from large vision-language models to encode
semantics in 3D representations. However, these approaches tend to produce maps
with per-point feature vectors, which do not scale well in larger environments,
nor do they contain semantic spatial relationships between entities in the
environment, which are useful for downstream planning. In this work, we propose
ConceptGraphs, an open-vocabulary graph-structured representation for 3D
scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
their output to 3D by multi-view association. The resulting representations
generalize to novel semantic classes, without the need to collect large 3D
datasets or finetune models. We demonstrate the utility of this representation
through a number of downstream planning tasks that are specified through
abstract (language) prompts and require complex reasoning over spatial and
semantic concepts. (Project page: this https URL Explainer
video: this https URL )

为了使机器人能够执行各种任务，本研究提出了一种名为 ConceptGraphs 的基于图结构的 3D 场景表示方法，通过将 2D 基本模型的输出与多视图关联相结合，不需要收集大型 3D 数据集或微调模型即可实现对新的语义类进行泛化推理，并通过一些需要对空间和语义概念进行复杂推理的后续规划任务来证明这种表示方法的实用性。