Open-vocabulary segmentation of 3D scenes is a fundamental function of human
perception and thus a crucial objective in computer vision research. However,
this task is heavily impeded by the lack of large-scale and diverse 3D
open-vocabulary segmentation datasets for training robust and generalizable
models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation
models helps but it compromises the open-vocabulary feature significantly as
the 2D models are mostly finetuned with close-vocabulary datasets. We tackle
the challenges in 3D open-vocabulary segmentation by exploiting the
open-vocabulary multimodal knowledge and object reasoning capability of
pre-trained foundation models CLIP and DINO, without necessitating any
fine-tuning. Specifically, we distill open-vocabulary visual and textual
knowledge from CLIP into a neural radiance field (NeRF) which effectively lifts
2D features into view-consistent 3D segmentation. Furthermore, we introduce the
Relevancy-Distribution Alignment loss and Feature-Distribution Alignment loss
to respectively mitigate the ambiguities of CLIP features and distill precise
object boundaries from DINO features, eliminating the need for segmentation
annotations during training. Extensive experiments show that our method even
outperforms fully supervised models trained with segmentation annotations,
suggesting that 3D open-vocabulary segmentation can be effectively learned from
2D images and text-image pairs.

本研究利用预训练的 CLIP 和 DINO 模型的多模式知识和物体推理能力来解决三维开放式词汇分割的挑战。通过优化神经辐射场 (NeRF) 并引入相应的损失函数，本研究在无需进行微调的情况下，以开放式视觉和文本知识为先验知识，从 2D 特征中提取出 3D 分割特征。通过实验验证，本研究所提出的方法在无需分割注释的情况下，甚至优于完全监督的分割模型训练，表明三维开放式词汇分割可以有效地从 2D 图像和文本图像对中学到。