3D semantic scene understanding is a fundamental challenge in computer
vision. It enables mobile agents to autonomously plan and navigate arbitrary
environments. SSC formalizes this challenge as jointly estimating dense
geometry and semantic information from sparse observations of a scene. Current
methods for SSC are generally trained on 3D ground truth based on aggregated
LiDAR scans. This process relies on special sensors and annotation by hand
which are costly and do not scale well. To overcome this issue, our work
presents the first self-supervised approach to SSC called S4C that does not
rely on 3D ground truth data. Our proposed method can reconstruct a scene from
a single image and only relies on videos and pseudo segmentation ground truth
generated from off-the-shelf image segmentation network during training. Unlike
existing methods, which use discrete voxel grids, we represent scenes as
implicit semantic fields. This formulation allows querying any point within the
camera frustum for occupancy and semantic class. Our architecture is trained
through rendering-based self-supervised losses. Nonetheless, our method
achieves performance close to fully supervised state-of-the-art methods.
Additionally, our method demonstrates strong generalization capabilities and
can synthesize accurate segmentation maps for far away viewpoints.

我们提出了一种名为 S4C 的第一种自我监督方法，不依赖于 3D 完整数据，能够从单张图像中重建场景，并且只依赖于视频和训练期间从现成图像分割网络生成的伪分割地面实况。与现有方法不同，我们将场景表示为隐式语义场，通过渲染为基础的自我监督损失训练我们的架构，取得了接近完全监督最新方法的性能，并且表现出强大的泛化能力，可以为远距离视点合成准确的分割地图。