Existing 3D scene understanding tasks have achieved high performance on
close-set benchmarks but fail to handle novel categories in real-world
applications. To this end, we propose a Regional Point-Language Contrastive
learning framework, namely RegionPLC, for open-world 3D scene understanding,
which equips models trained on closed-set datasets with open-vocabulary
recognition capabilities. We propose dense visual prompts to elicit
region-level visual-language knowledge from 2D foundation models via
captioning, which further allows us to build dense regional point-language
associations. Then, we design a point-discriminative contrastive learning
objective to enable point-independent learning from captions for dense scene
understanding. We conduct extensive experiments on ScanNet, ScanNet200, and
nuScenes datasets. Our RegionPLC significantly outperforms previous
base-annotated 3D open-world scene understanding approaches by an average of
11.6\% and 6.6\% for semantic and instance segmentation, respectively. It also
shows promising open-world results in absence of any human annotation with low
training and inference costs. Code will be released.

本文提出了一种开放式 3D 场景理解模型 ——RegionPLC，并通过对模型进行训练，使其具有开放词汇识别能力。采用密集视觉提示来引导从 2D 基本模型中提取区域级视觉语言知识，通过指针鉴别式对比学习，进一步使其实现从标题中对密集场景进行理解，最终在 ScanNet、ScanNet200 和 nuScenes 数据集上进行了详尽实验，实现的结果较之前的 3D 开放式场景理解方法有显著提升。