We introduce SAMPro3D for zero-shot 3D indoor scene segmentation. Given the
3D point cloud and multiple posed 2D frames of 3D scenes, our approach segments
3D scenes by applying the pretrained Segment Anything Model (SAM) to 2D frames.
Our key idea involves locating 3D points in scenes as natural 3D prompts to
align their projected pixel prompts across frames, ensuring frame-consistency
in both pixel prompts and their SAM-predicted masks. Moreover, we suggest
filtering out low-quality 3D prompts based on feedback from all 2D frames, for
enhancing segmentation quality. We also propose to consolidate different 3D
prompts if they are segmenting the same object, bringing a more comprehensive
segmentation. Notably, our method does not require any additional training on
domain-specific data, enabling us to preserve the zero-shot power of SAM.
Extensive qualitative and quantitative results show that our method
consistently achieves higher quality and more diverse segmentation than
previous zero-shot or fully supervised approaches, and in many cases even
surpasses human-level annotations. The project page can be accessed at
this https URL

我们引入了 SAMPro3D 来进行零样本 3D 室内场景分割，通过将预训练的 Segment Anything Model (SAM) 应用于 2D 帧，在 3D 点云和多个姿态的 2D 帧的基础上对 3D 场景进行分割。我们的关键思想是将场景中的 3D 点作为自然的 3D 提示，对其在帧间的投影像素提示进行对齐，确保像素提示和其通过 SAM 预测的掩码在帧间的一致性。此外，我们建议根据所有 2D 帧的反馈滤除低质量的 3D 提示，以提高分割质量。我们还建议，如果不同的 3D 提示正在分割相同的对象，要进行融合，实现更全面的分割。值得注意的是，我们的方法不需要在领域特定数据上进行额外的训练，从而保留了 SAM 的零样本能力。广泛的定性和定量结果显示，我们的方法始终比先前的零样本或完全监督方法具有更高的质量和更多样的分割性能，在许多情况下甚至超过人类级别的注释。项目页面可以通过此 https URL 访问。