Existing 3D scene understanding tasks have achieved high performance on close-set benchmarks but fail to handle novel categories in real-world applications. To this end, we propose a Regional Point-Language Contrastive learning framework, namely RegionPLC, for open-world 3D scene understanding, which equips models trained on closed-set datasets with open-vocabulary recognition capabilities. We propose dense visual prompts to elicit region-level visual-language knowledge from 2D foundation models via captioning, which further allows us to build dense regional point-language associations. Then, we design a point-discriminative contrastive learning objective to enable point-independent learning from captions for dense scene understanding. We conduct extensive experiments on ScanNet, ScanNet200, and nuScenes datasets. Our RegionPLC significantly outperforms previous base-annotated 3D open-world scene understanding approaches by an average of 11.6\% and 6.6\% for semantic and instance segmentation, respectively. It also shows promising open-world results in absence of any human annotation with low training and inference costs. Code will be released.

本文提出了一种开放式3D场景理解模型——RegionPLC，并通过对模型进行训练，使其具有开放词汇识别能力。采用密集视觉提示来引导从2D基本模型中提取区域级视觉语言知识，通过指针鉴别式对比学习，进一步使其实现从标题中对密集场景进行理解，最终在ScanNet、ScanNet200和nuScenes数据集上进行了详尽实验，实现的结果较之前的3D开放式场景理解方法有显著提升。

RegionPLC：区域点-语言对比学习用于开放世界3D场景理解