Applying NeRF to downstream perception tasks for scene understanding and
representation is becoming increasingly popular. Most existing methods treat
semantic prediction as an additional rendering task, \textit{i.e.}, the "label
rendering" task, to build semantic NeRFs. However, by rendering
semantic/instance labels per pixel without considering the contextual
information of the rendered image, these methods usually suffer from unclear
boundary segmentation and abnormal segmentation of pixels within an object. To
solve this problem, we propose Generalized Perception NeRF (GP-NeRF), a novel
pipeline that makes the widely used segmentation model and NeRF work compatibly
under a unified framework, for facilitating context-aware 3D scene perception.
To accomplish this goal, we introduce transformers to aggregate radiance as
well as semantic embedding fields jointly for novel views and facilitate the
joint volumetric rendering of both fields. In addition, we propose two
self-distillation mechanisms, i.e., the Semantic Distill Loss and the
Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality
of the semantic field and the maintenance of geometric consistency. In
evaluation, we conduct experimental comparisons under two perception tasks
(\textit{i.e.} semantic and instance segmentation) using both synthetic and
real-world datasets. Notably, our method outperforms SOTA approaches by 6.94\%,
11.76\%, and 8.47\% on generalized semantic segmentation, finetuning semantic
segmentation, and instance segmentation, respectively.

提出了一种通用感知 NeRF（GP-NeRF）模型，通过引入 transformers 和自我蒸馏机制，实现了语义场和几何场的联合渲染，以促进具有上下文感知的三维场景理解。在评估中，使用合成和真实世界数据集进行了两个感知任务（即语义和实例分割）的实验比较，取得了显著的优于现有方法的效果。