3D semantic scene understanding is a fundamental challenge in computer
vision. It enables mobile agents to autonomously plan and navigate arbitrary
environments. SSC formalizes this challenge as jointly estimating dense
geometry and semantic information from sparse observations of a scene. Current
methods for SSC are generally trained on 3D ground truth based on aggregated
LiDAR scans. This process relies on special sensors and annotation by hand
which are costly and do not scale well. To overcome this issue, our work
presents the first self-supervised approach to SSC called S4C that does not
rely on 3D ground truth data. Our proposed method can reconstruct a scene from
a single image and only relies on videos and pseudo segmentation ground truth
generated from off-the-shelf image segmentation network during training. Unlike
existing methods, which use discrete voxel grids, we represent scenes as
implicit semantic fields. This formulation allows querying any point within the
camera frustum for occupancy and semantic class. Our architecture is trained
through rendering-based self-supervised losses. Nonetheless, our method
achieves performance close to fully supervised state-of-the-art methods.
Additionally, our method demonstrates strong generalization capabilities and
can synthesize accurate segmentation maps for far away viewpoints.

我们提出了一种名为 S4C 的第一种自我监督方法，不依赖于 3D 完整数据，能够从单张图像中重建场景，并且只依赖于视频和训练期间从现成图像分割网络生成的伪分割地面实况。与现有方法不同，我们将场景表示为隐式语义场，通过渲染为基础的自我监督损失训练我们的架构，取得了接近完全监督最新方法的性能，并且表现出强大的泛化能力，可以为远距离视点合成准确的分割地图。

S4C：自我监督的神经场景补全

S4C: Self-Supervised Semantic Scene Completion with Neural Fields

We present ScanNet++, a large-scale dataset that couples together capture of
high-quality and commodity-level geometry and color of indoor scenes. Each
scene is captured with a high-end laser scanner at sub-millimeter resolution,
along with registered 33-megapixel images from a DSLR camera, and RGB-D streams
from an iPhone. Scene reconstructions are further annotated with an open
vocabulary of semantics, with label-ambiguous scenarios explicitly annotated
for comprehensive semantic understanding. ScanNet++ enables a new real-world
benchmark for novel view synthesis, both from high-quality RGB capture, and
importantly also from commodity-level images, in addition to a new benchmark
for 3D semantic scene understanding that comprehensively encapsulates diverse
and ambiguous semantic labeling scenarios. Currently, ScanNet++ contains 460
scenes, 280,000 captured DSLR images, and over 3.7M iPhone RGBD frames.

ScanNet++ 是一个大规模数据集，结合高质量和大众级几何和室内场景的颜色采集。它包括使用高端激光扫描仪以亚毫米的分辨率捕捉的场景，以及来自 DSLR 相机的 3300 万像素注册图像和 iPhone 的 RGB-D 流。该数据集为新视角合成提供了一个新的实际世界基准，不仅包括高质量的 RGB 采集，还包括大众级图像，同时也提供了一个全面涵盖各种模糊语义标注场景的三维语义场景理解基准。目前，ScanNet++ 包含 460 个场景，280,000 张 DSLR 图像和超过 3.7M 个 iPhone 的 RGBD 帧。

ScanNet++: 一个高保真度的三维室内场景数据集

ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

3D semantic scene understanding tasks have achieved great success with the
emergence of deep learning, but often require a huge amount of manually
annotated training data. To alleviate the annotation cost, we propose the first
weakly-supervised 3D instance segmentation method that only requires
categorical semantic labels as supervision, and we do not need instance-level
labels. The required semantic annotations can be either dense or extreme sparse
(e.g. 0.02% of total points). Even without having any instance-related
ground-truth, we design an approach to break point clouds into raw fragments
and find the most confident samples for learning instance centroids.
Furthermore, we construct a recomposed dataset using pseudo instances, which is
used to learn our defined multilevel shape-aware objectness signal. An
asymmetrical object inference algorithm is followed to process core points and
boundary points with different strategies, and generate high-quality pseudo
instance labels to guide iterative training. Experiments demonstrate that our
method can achieve comparable results with recent fully supervised methods. By
generating pseudo instance labels from categorical semantic labels, our
designed approach can also assist existing methods for learning 3D instance
segmentation at reduced annotation cost.

通过使用弱监督的三维实例分割方法，只需要分类语义标签作为监督，而无需实例级别标签，从而减少批注成本，并利用伪实例标签辅助现有方法学习三维实例分割。