Semantic labelling is highly correlated with geometry and radiance
reconstruction, as scene entities with similar shape and appearance are more
likely to come from similar classes. Recent implicit neural reconstruction
techniques are appealing as they do not require prior training data, but the
same fully self-supervised approach is not possible for semantics because
labels are human-defined properties.
We extend neural radiance fields (NeRF) to jointly encode semantics with
appearance and geometry, so that complete and accurate 2D semantic labels can
be achieved using a small amount of in-place annotations specific to the scene.
The intrinsic multi-view consistency and smoothness of NeRF benefit semantics
by enabling sparse labels to efficiently propagate. We show the benefit of this
approach when labels are either sparse or very noisy in room-scale scenes. We
demonstrate its advantageous properties in various interesting applications
such as an efficient scene labelling tool, novel semantic view synthesis, label
denoising, super-resolution, label interpolation and multi-view semantic label
fusion in visual semantic mapping systems.

本研究旨在扩展神经辐射场（NeRF）方法，并通过将语义信息与外观和几何信息进行联合编码，实现同时获取精确的 2D 语义标签，而无需使用过多的现场注释数据，该方法通过有效地传播稀疏或嘈杂的标签，在轻松地应用于室内场景标注、超分辨率、标签插值和多视图语义标签融合等多个有趣应用展示了其优越性。