We propose PanopticFusion, a novel online volumetric semantic mapping system
at the level of stuff and things. In contrast to previous semantic mapping
systems, PanopticFusion is able to densely predict class labels of a background
region (stuff) and individually segment arbitrary foreground objects (things).
In addition, our system has the capability to reconstruct a large-scale scene
and extract a labeled mesh thanks to its use of a spatially hashed volumetric
map representation. Our system first predicts pixel-wise panoptic labels (class
labels for stuff regions and instance IDs for thing regions) for incoming RGB
frames by fusing 2D semantic and instance segmentation outputs. The predicted
panoptic labels are integrated into the volumetric map together with depth
measurements while keeping the consistency of the instance IDs, which could
vary frame to frame, by referring to the 3D map at that moment. In addition, we
construct a fully connected conditional random field (CRF) model with respect
to panoptic labels for map regularization. For online CRF inference, we propose
a novel unary potential approximation and a map division strategy.
We evaluated the performance of our system on the ScanNet (v2) dataset.
PanopticFusion outperformed or compared with state-of-the-art offline 3D DNN
methods in both semantic and instance segmentation benchmarks. Also, we
demonstrate a promising augmented reality application using a 3D panoptic map
generated by the proposed system.

PanopticFusion 是一种新颖的在线立体语义映射系统，能够在 stuff 和 things 的级别上密集预测背景区域（stuff）的类标签和不同前景物体（things）的各自分割，利用其空间哈希体积图表示可以重构大规模场景并提取标记的网格，通过融合 2D 语义和实例分割输出对 RGB 框架的像素级 Panoptic 标签进行首次预测，并将这些标签与深度测量一起整合到体积地图中进行 CRF 模型的规范化，通过提出的一种新的一次势近似和地图分割策略实现在线 CRF 推理，我们在 ScanNet（v2）数据集上评估了系统性能，发现其在语义和实例分割基准测试中均优于或与最先进的脱机 3D DNN 方法相比，同时演示了利用该系统生成的 3D Panoptic 地图的有前途的增强现实应用。