Building 3D maps of the environment is central to robot navigation, planning,
and interaction with objects in a scene. Most existing approaches that
integrate semantic concepts with 3D maps largely remain confined to the
closed-set setting: they can only reason about a finite set of concepts,
pre-defined at training time. Further, these maps can only be queried using
class labels, or in recent work, using text prompts.
We address both these issues with ConceptFusion, a scene representation that
is (1) fundamentally open-set, enabling reasoning beyond a closed set of
concepts and (ii) inherently multimodal, enabling a diverse range of possible
queries to the 3D map, from language, to images, to audio, to 3D geometry, all
working in concert. ConceptFusion leverages the open-set capabilities of
today's foundation models pre-trained on internet-scale data to reason about
concepts across modalities such as natural language, images, and audio. We
demonstrate that pixel-aligned open-set features can be fused into 3D maps via
traditional SLAM and multi-view fusion approaches. This enables effective
zero-shot spatial reasoning, not needing any additional training or finetuning,
and retains long-tailed concepts better than supervised approaches,
outperforming them by more than 40% margin on 3D IoU. We extensively evaluate
ConceptFusion on a number of real-world datasets, simulated home environments,
a real-world tabletop manipulation task, and an autonomous driving platform. We
showcase new avenues for blending foundation models with 3D open-set multimodal
mapping.
For more information, visit our project page this https URL
or watch our 5-minute explainer video
this https URL

ConceptFusion 是一种基本的开放式多模态场景表示方法，通过集成预先训练的基础模型的能力，可跨越自然语言、图像和音频等多种模式推理和查询，从而实现零训练空间推理，并在保留长尾概念方面表现比监督方法优异 40% 的跨模态三维映射。