Recently, methods have been proposed for 3D open-vocabulary semantic
segmentation. Such methods are able to segment scenes into arbitrary classes
given at run-time using their text description. In this paper, we propose to
our knowledge the first algorithm for open-vocabulary panoptic segmentation,
simultaneously performing both semantic and instance segmentation. Our
algorithm, Panoptic Vision-Language Feature Fields (PVLFF) learns a feature
field of the scene, jointly learning vision-language features and hierarchical
instance features through a contrastive loss function from 2D instance segment
proposals on input frames. Our method achieves comparable performance against
the state-of-the-art close-set 3D panoptic systems on the HyperSim, ScanNet and
Replica dataset and outperforms current 3D open-vocabulary systems in terms of
semantic segmentation. We additionally ablate our method to demonstrate the
effectiveness of our model architecture. Our code will be available at
this https URL

我们提出了一种新的算法，Panoptic Vision-Language Feature Fields (PVLFF)，可以同时进行语义和实例分割，通过对输入帧上的 2D 实例分割提案应用对比损失函数来联合学习视觉 - 语言特征和分层实例特征，从而在 HyperSim、ScanNet 和 Replica 数据集上达到可比较的性能并在语义分割方面优于当前的 3D 开放词汇系统。