We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for
grounding a variety of entities, such as object instances, agents, and regions,
with free-form text-based queries. Unlike conventional semantic-based object
localization approaches, our system facilitates context-aware entity
localization, allowing for queries such as ``pick up a cup on a kitchen table"
or ``navigate to a sofa on which someone is sitting". In contrast to existing
research on 3D scene graphs, OVSG supports free-form text input and
open-vocabulary querying. Through a series of comparative experiments using the
ScanNet dataset and a self-collected dataset, we demonstrate that our proposed
approach significantly surpasses the performance of previous semantic-based
localization techniques. Moreover, we highlight the practical application of
OVSG in real-world robot navigation and manipulation experiments.

我们介绍了一种开放词汇的 3D 场景图（OVSG），它是一个形式化的框架，用于将各种实体，如物体实例、代理和区域，与自由文本查询进行关联。与传统的基于语义的物体定位方法不同，我们的系统支持上下文感知的实体定位，允许查询，如 ' 在厨房桌子上拿起一个杯子 ' 或' 导航到一个有人坐在上面的沙发 '。与现有的 3D 场景图研究相比，OVSG 支持自由文本输入和开放词汇查询。通过使用 ScanNet 数据集和自采集数据集进行的一系列比较实验证明，我们提出的方法明显超越了以前基于语义的定位技术的性能。此外，我们突出了 OVSG 在真实世界的机器人导航和操作实验中的实际应用。

基于上下文感知的实体定位和开放词汇 3D 场景图

Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs

Natural language is one of the most intuitive ways to express human intent.
However, translating instructions and commands towards robotic motion
generation and deployment in the real world is far from being an easy task. The
challenge of combining a robot's inherent low-level geometric and kinodynamic
constraints with a human's high-level semantic instructions traditionally is
solved using task-specific solutions with little generalizability between
hardware platforms, often with the use of static sets of target actions and
commands. This work instead proposes a flexible language-based framework that
allows a user to modify generic robotic trajectories. Our method leverages
pre-trained language models (BERT and CLIP) to encode the user's intent and
target objects directly from a free-form text input and scene images, fuses
geometrical features generated by a transformer encoder network, and finally
outputs trajectories using a transformer decoder, without the need of priors
related to the task or robot information. We significantly extend our own
previous work presented in Bucker et al. by expanding the trajectory
parametrization space to 3D and velocity as opposed to just XY movements. In
addition, we now train the model to use actual images of the objects in the
scene for context (as opposed to textual descriptions), and we evaluate the
system in a diverse set of scenarios beyond manipulation, such as aerial and
legged robots. Our simulated and real-life experiments demonstrate that our
transformer model can successfully follow human intent, modifying the shape and
speed of trajectories within multiple environments. Codebase available at:
this https URL

该研究提出了一个灵活的基于语言的框架，使用自然语言输入和场景图像处理机器人运动指令，通过预先训练的语言模型和变压器编码器网络生成目标几何特征，然后使用变压器解码器生成轨迹，无需任何先验知识。在复杂环境下，仿真和实际实验表明，该系统可以成功地跟随人类意图，修改轨迹的形状和速度。