Current approaches for 3D scene graph prediction rely on labeled datasets to
train models for a fixed set of known object classes and relationship
categories. We present Open3DSG, an alternative approach to learn 3D scene
graph prediction in an open world without requiring labeled scene graph data.
We co-embed the features from a 3D scene graph prediction backbone with the
feature space of powerful open world 2D vision language foundation models. This
enables us to predict 3D scene graphs from 3D point clouds in a zero-shot
manner by querying object classes from an open vocabulary and predicting the
inter-object relationships from a grounded LLM with scene graph features and
queried object classes as context. Open3DSG is the first 3D point cloud method
to predict not only explicit open-vocabulary object classes, but also open-set
relationships that are not limited to a predefined label set, making it
possible to express rare as well as specific objects and relationships in the
predicted 3D scene graph. Our experiments show that Open3DSG is effective at
predicting arbitrary object classes as well as their complex inter-object
relationships describing spatial, supportive, semantic and comparative
relationships.

使用开放的词汇表以及与场景图特征和查询对象类别联系的扎根 LLM，Open3DSG 作为第一个可以预测不仅是明确的开放词汇对象类别，还包括预定义标签集以外的开放集关系的三维点云方法，在零样本学习方式下有效预测任意对象类别及其描述空间、支持、语义和比较关系的复杂对象之间的关系。