Natural language interfaces to embodied AI are becoming more ubiquitous in
our daily lives. This opens further opportunities for language-based
interaction with embodied agents, such as a user instructing an agent to
execute some task in a specific location. For example, "put the bowls back in
the cupboard next to the fridge" or "meet me at the intersection under the red
sign." As such, we need methods that interface between natural language and map
representations of the environment. To this end, we explore the question of
whether we can use an open-set natural language query to identify a scene
represented by a 3D scene graph. We define this task as "language-based
scene-retrieval" and it is closely related to "coarse-localization," but we are
instead searching for a match from a collection of disjoint scenes and not
necessarily a large-scale continuous map. Therefore, we present
Text2SceneGraphMatcher, a "scene-retrieval" pipeline that learns joint
embeddings between text descriptions and scene graphs to determine if they are
matched. The code, trained models, and datasets will be made public.

使用自然语言接口和语境 AI 的研究，探索以自然语言查询来识别 3D 场景图的场景检索方法，并介绍了 Text2SceneGraphMatcher 的学习框架和数据集。

语言辅助下的场景检索：我在哪里？

"Where am I?" Scene Retrieval with Language

We devise a 3D scene graph representation, contact graph+ (cg+), for
efficient sequential task planning. Augmented with predicate-like attributes,
this contact graph-based representation abstracts scene layouts with succinct
geometric information and valid robot-scene interactions. Goal configurations,
naturally specified on contact graphs, can be produced by a genetic algorithm
with a stochastic optimization method. A task plan is then initialized by
computing the Graph Editing Distance (GED) between the initial contact graphs
and the goal configurations, which generates graph edit operations
corresponding to possible robot actions. We finalize the task plan by imposing
constraints to regulate the temporal feasibility of graph edit operations,
ensuring valid task and motion correspondences. In a series of simulations and
experiments, robots successfully complete complex sequential object
rearrangement tasks that are difficult to specify using conventional planning
language like Planning Domain Definition Language (PDDL), demonstrating the
high feasibility and potential of robot sequential task planning on contact
graph.

本文提出了一种基于 3D 场景图表示的联系图 +（CG +）的机器人规划方法，利用谓词属性增强这种联系图表示，进行任务规划，计算初始场景图和目标配置之间的图形编辑距离（GED）来初始化任务计划，最终通过实施约束来完成任务，并通过系列实验验证其潜力和可行性。

场景图上的顺序操纵规划

Sequential Manipulation Planning on Scene Graph

In this paper, we propose a novel Knowledge-based Embodied Question Answering
(K-EQA) task, in which the agent intelligently explores the environment to
answer various questions with the knowledge. Different from explicitly
specifying the target object in the question as existing EQA work, the agent
can resort to external knowledge to understand more complicated question such
as "Please tell me what are objects used to cut food in the room?", in which
the agent must know the knowledge such as "knife is used for cutting food".
To address this K-EQA problem, a novel framework based on neural program
synthesis reasoning is proposed, where the joint reasoning of the external
knowledge and 3D scene graph is performed to realize navigation and question
answering. Especially, the 3D scene graph can provide the memory to store the
visual information of visited scenes, which significantly improves the
efficiency for the multi-turn question answering. Experimental results have
demonstrated that the proposed framework is capable of answering more
complicated and realistic questions in the embodied environment. The proposed
method is also applicable to multi-agent scenarios.

本文提出了一种基于知识的具身问答 (Knowledge-based Embodied Question Answering, K-EQA) 任务，其中智能体通过利用知识来探索环境以回答各种问题。为解决这一问题，本文提出了一种基于神经程序合成推理的新框架，同时进行外部知识和 3D 场景图的联合推理，以实现导航和问答。实验结果表明，该框架能够在具身环境中回答更加复杂和现实的问题，并且该方法也适用于多智能体场景。