Current approaches for 3D scene graph prediction rely on labeled datasets to
train models for a fixed set of known object classes and relationship
categories. We present Open3DSG, an alternative approach to learn 3D scene
graph prediction in an open world without requiring labeled scene graph data.
We co-embed the features from a 3D scene graph prediction backbone with the
feature space of powerful open world 2D vision language foundation models. This
enables us to predict 3D scene graphs from 3D point clouds in a zero-shot
manner by querying object classes from an open vocabulary and predicting the
inter-object relationships from a grounded LLM with scene graph features and
queried object classes as context. Open3DSG is the first 3D point cloud method
to predict not only explicit open-vocabulary object classes, but also open-set
relationships that are not limited to a predefined label set, making it
possible to express rare as well as specific objects and relationships in the
predicted 3D scene graph. Our experiments show that Open3DSG is effective at
predicting arbitrary object classes as well as their complex inter-object
relationships describing spatial, supportive, semantic and comparative
relationships.

使用开放的词汇表以及与场景图特征和查询对象类别联系的扎根 LLM，Open3DSG 作为第一个可以预测不仅是明确的开放词汇对象类别，还包括预定义标签集以外的开放集关系的三维点云方法，在零样本学习方式下有效预测任意对象类别及其描述空间、支持、语义和比较关系的复杂对象之间的关系。

Open3DSG: 从点云生成可查询对象和开放关系的开放式词汇 3D 场景图

Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with  Queryable Objects and Open-Set Relationships

3D scene graph prediction is a task that aims to concurrently predict object
classes and their relationships within a 3D environment. As these environments
are primarily designed by and for humans, incorporating commonsense knowledge
regarding objects and their relationships can significantly constrain and
enhance the prediction of the scene graph. In this paper, we investigate the
application of commonsense knowledge graphs for 3D scene graph prediction on
point clouds of indoor scenes. Through experiments conducted on a real-world
indoor dataset, we demonstrate that integrating external commonsense knowledge
via the message-passing method leads to a 15.0 % improvement in scene graph
prediction accuracy with external knowledge and $7.96\%$ with internal
knowledge when compared to state-of-the-art algorithms. We also tested in the
real world with 10 frames per second for scene graph generation to show the
usage of the model in a more realistic robotics setting.

应用常识知识图谱以提高室内场景点云的 3D 场景图预测准确度的实验研究表明，在外部常识知识和内部知识的交互应用中，通过消息传递的方法比最先进的算法提高了 15.0% 的场景图预测准确度和 7.96% 的内部知识，在每秒 10 帧的实际机器人环境中也进行了测试以展示模型的使用情况。