In this paper, we address the challenge of decomposing Neural Radiance Fields (NeRF) into objects from an open vocabulary, a critical task for object manipulation in 3D reconstruction and view synthesis. Current techniques for NeRF decomposition involve a trade-off between the flexibility of processing open-vocabulary queries and the accuracy of 3D segmentation. We present, Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), that leverage large-scale, off-the-shelf, segmentation models like the Segment Anything Model (SAM) and introduce an integrate-and-distill paradigm with hierarchical embeddings to achieve both the flexibility of open-vocabulary querying and 3D segmentation accuracy. Open-NeRF first utilizes large-scale foundation models to generate hierarchical 2D mask proposals from varying viewpoints. These proposals are then aligned via tracking approaches and integrated within the 3D space and subsequently distilled into the 3D field. This process ensures consistent recognition and granularity of objects from different viewpoints, even in challenging scenarios involving occlusion and indistinct features. Our experimental results show that the proposed Open-NeRF outperforms state-of-the-art methods such as LERF \cite{lerf} and FFD \cite{ffd} in open-vocabulary scenarios. Open-NeRF offers a promising solution to NeRF decomposition, guided by open-vocabulary queries, enabling novel applications in robotics and vision-language interaction in open-world 3D scenes.

该研究关注解决从开放词汇中将神经辐射场（NeRF）分解为对象的挑战，这对于三维重建和视图合成中的对象操作是至关重要的。我们提出了Open-NeRF，利用大规模、现成的分割模型，如Segment Anything Model（SAM），并引入了一种集成和蒸馏范式，通过层次嵌入来既实现开放词汇查询的灵活性，又保持三维分割的准确性。Open-NeRF首先利用大规模的基础模型根据不同视角生成分层的二维掩模提案，然后通过跟踪方法对这些提案进行对齐，并在三维空间中进行集成，最后蒸馏为三维场。该过程确保了不同视角下的对象一致识别和细粒度，即使在涉及遮挡和模糊特征的挑战性场景中也是如此。实验结果表明，Open-NeRF在开放词汇的场景中胜过了LERF和FFD等最先进的方法。Open-NeRF为NeRF分解提供了一个有前途的解决方案，通过开放词汇查询引导，能够在开放世界的三维场景中实现新的机器人和视觉语言交互应用。

Open-NeRF：面向开放词汇的 NeRF 分解