D scene graphs are an emerging 3D scene representation, that models both the
objects present in the scene as well as their relationships. However, learning
3D scene graphs is a challenging task because it requires not only object
labels but also relationship annotations, which are very scarce in datasets.
While it is widely accepted that pre-training is an effective approach to
improve model performance in low data regimes, in this paper, we find that
existing pre-training methods are ill-suited for 3D scene graphs. To solve this
issue, we present the first language-based pre-training approach for 3D scene
graphs, whereby we exploit the strong relationship between scene graphs and
language. To this end, we leverage the language encoder of CLIP, a popular
vision-language model, to distill its knowledge into our graph-based network.
We formulate a contrastive pre-training, which aligns text embeddings of
relationships (subject-predicate-object triplets) and predicted 3D graph
features. Our method achieves state-of-the-art results on the main semantic 3D
scene graph benchmark by showing improved effectiveness over pre-training
baselines and outperforming all the existing fully supervised scene graph
prediction methods by a significant margin. Furthermore, since our scene graph
features are language-aligned, it allows us to query the language space of the
features in a zero-shot manner. In this paper, we show an example of utilizing
this property of the features to predict the room type of a scene without
further training.

我们提出了第一个针对 3D 场景图的基于语言的预训练方法，利用了场景图和语言之间的强关联，通过对比性预训练将文本嵌入关系和预测的 3D 图特征进行对齐，最终实现了在主要语义 3D 场景图基准上的效果最优，超过现有全监督场景图预测方法的显著边缘，并且由于场景图特征与语言对齐，允许我们以零样本的方式查询特征的语言空间。