This paper presents one of the first learning-based NeRF 3D instance
segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance
NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance
NeRF can learn 3D instance segmentation of a given scene, represented as an
instance field component of the NeRF model. To this end, we adopt a 3D
proposal-based mask prediction network on the sampled volumetric features from
NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction
is then projected to image space to match 2D segmentation masks from different
views generated by existing panoptic segmentation models, which are used to
supervise the training of the instance field. Notably, beyond generating
consistent 2D segmentation maps from novel views, Instance NeRF can query
instance information at any 3D point, which greatly enhances NeRF object
segmentation and manipulation. Our method is also one of the first to achieve
such results without ground-truth instance information during inference.
Experimented on synthetic and real-world NeRF datasets with complex indoor
scenes, Instance NeRF surpasses previous NeRF segmentation works and
competitive 2D segmentation methods in segmentation performance on unseen
views. See the demo video at this https URL

这篇论文介绍了一种名为 Instance NeRF 的学习型 NeRF 3D 实例分割管道，它采用 3D 提案基于掩膜预测网络，可以学习给定场景的 3D 实例分割，并在任意 3D 点查询实例信息，超越了以前的 NeRF 分割方法和具有竞争力的 2D 分割方法。