The Segment Anything Model (SAM) has demonstrated its effectiveness in
segmenting any object/part in various 2D images, yet its ability for 3D has not
been fully explored. The real world is composed of numerous 3D scenes and
objects. Due to the scarcity of accessible 3D data and high cost of its
acquisition and annotation, lifting SAM to 3D is a challenging but valuable
research avenue. With this in mind, we propose a novel framework to Segment
Anything in 3D, named SA3D. Given a neural radiance field (NeRF) model, SA3D
allows users to obtain the 3D segmentation result of any target object via only
one-shot manual prompting in a single rendered view. With input prompts, SAM
cuts out the target object from the according view. The obtained 2D
segmentation mask is projected onto 3D mask grids via density-guided inverse
rendering. 2D masks from other views are then rendered, which are mostly
uncompleted but used as cross-view self-prompts to be fed into SAM again.
Complete masks can be obtained and projected onto mask grids. This procedure is
executed via an iterative manner while accurate 3D masks can be finally
learned. SA3D can adapt to various radiance fields effectively without any
additional redesigning. The entire segmentation process can be completed in
approximately two minutes without any engineering optimization. Our experiments
demonstrate the effectiveness of SA3D in different scenes, highlighting the
potential of SAM in 3D scene perception. The project page is at
this https URL

该研究提出了一个用于三维分割的新框架 SA3D，通过密度导向的反渲染将 2D 分割蒙版投影到 3D 蒙版网格，再通过迭代的方式学习准确的 3D 分割结果，实验表明 SA3D 在不同场景下的有效性。