Recently, large-scale pre-trained models such as Segment-Anything Model (SAM)
and Contrastive Language-Image Pre-training (CLIP) have demonstrated remarkable
success and revolutionized the field of computer vision. These foundation
vision models effectively capture knowledge from a large-scale broad data with
their vast model parameters, enabling them to perform zero-shot segmentation on
previously unseen data without additional training. While they showcase
competence in 2D tasks, their potential for enhancing 3D scene understanding
remains relatively unexplored. To this end, we present a novel framework that
adapts various foundational models for the 3D point cloud segmentation task.
Our approach involves making initial predictions of 2D semantic masks using
different large vision models. We then project these mask predictions from
various frames of RGB-D video sequences into 3D space. To generate robust 3D
semantic pseudo labels, we introduce a semantic label fusion strategy that
effectively combines all the results via voting. We examine diverse scenarios,
like zero-shot learning and limited guidance from sparse 2D point labels, to
assess the pros and cons of different vision foundation models. Our approach is
experimented on ScanNet dataset for 3D indoor scenes, and the results
demonstrate the effectiveness of adopting general 2D foundation models on
solving 3D point cloud segmentation tasks.

通过采用大规模预训练模型、计算机视觉和 2D 任务，本研究提出了一种新的框架来解决 3D 点云分割任务，通过将 2D 语义分割预测投影到 3D 空间，并引入语义标签融合策略，实现了在扩展的数据集上的 3D 场景理解。