Natural language interfaces to embodied AI are becoming more ubiquitous in
our daily lives. This opens further opportunities for language-based
interaction with embodied agents, such as a user instructing an agent to
execute some task in a specific location. For example, "put the bowls back in
the cupboard next to the fridge" or "meet me at the intersection under the red
sign." As such, we need methods that interface between natural language and map
representations of the environment. To this end, we explore the question of
whether we can use an open-set natural language query to identify a scene
represented by a 3D scene graph. We define this task as "language-based
scene-retrieval" and it is closely related to "coarse-localization," but we are
instead searching for a match from a collection of disjoint scenes and not
necessarily a large-scale continuous map. Therefore, we present
Text2SceneGraphMatcher, a "scene-retrieval" pipeline that learns joint
embeddings between text descriptions and scene graphs to determine if they are
matched. The code, trained models, and datasets will be made public.

使用自然语言接口和语境 AI 的研究，探索以自然语言查询来识别 3D 场景图的场景检索方法，并介绍了 Text2SceneGraphMatcher 的学习框架和数据集。