3D reconstruction has been widely used in autonomous navigation fields of
mobile robotics. However, the former research can only provide the basic
geometry structure without the capability of open-world scene understanding,
limiting advanced tasks like human interaction and visual navigation. Moreover,
traditional 3D scene understanding approaches rely on expensive labeled 3D
datasets to train a model for a single task with supervision. Thus, geometric
reconstruction with zero-shot scene understanding i.e. Open vocabulary 3D
Understanding and Reconstruction, is crucial for the future development of
mobile robots. In this paper, we propose OpenOcc, a novel framework unifying
the 3D scene reconstruction and open vocabulary understanding with neural
radiance fields. We model the geometric structure of the scene with occupancy
representation and distill the pre-trained open vocabulary model into a 3D
language field via volume rendering for zero-shot inference. Furthermore, a
novel semantic-aware confidence propagation (SCP) method has been proposed to
relieve the issue of language field representation degeneracy caused by
inconsistent measurements in distilled features. Experimental results show that
our approach achieves competitive performance in 3D scene understanding tasks,
especially for small and long-tail objects.

我们提出了 OpenOcc，一种将 3D 场景重建和开放词汇理解与神经辐射场结合的新型框架。通过占位表示法对场景的几何结构进行建模，并通过体素渲染将预训练的开放词汇模型蒸馏为 3D 语言场，以实现零 - shot 推理。此外，我们提出了一种新颖的语义感知置信度传播（SCP）方法，以缓解由于蒸馏特征中不一致测量而引起的语言场表示退化问题。实验证明，我们的方法在 3D 场景理解任务中取得了有竞争力的性能，特别是对于小物体和长尾物体。