We introduce 3D-SIS, a novel neural network architecture for 3D semantic
instance segmentation in commodity RGB-D scans. The core idea of our method is
to jointly learn from both geometric and color signal, thus enabling accurate
instance predictions. Rather than operate solely on 2D frames, we observe that
most computer vision applications have multi-view RGB-D input available, which
we leverage to construct an approach for 3D instance segmentation that
effectively fuses together these multi-modal inputs. Our network leverages
high-resolution RGB input by associating 2D images with the volumetric grid
based on the pose alignment of the 3D reconstruction. For each image, we first
extract 2D features for each pixel with a series of 2D convolutions; we then
backproject the resulting feature vector to the associated voxel in the 3D
grid. This combination of 2D and 3D feature learning allows significantly
higher accuracy object detection and instance segmentation than
state-of-the-art alternatives. We show results on both synthetic and real-world
public benchmarks, achieving an improvement in mAP of over 13 on real-world
data.

本文介绍了一种新型的神经网络体系结构：3D-SIS，用于在商品 RGB-D 扫描中进行三维语义实例分割。它结合了几何和颜色信号的联合学习，从而实现了精确的实例预测。该网络利用了高分辨率 RGB 输入和多视角 RGB-D 输入，并在 3D 重建的姿态对准的基础上将 2D 图像与体积网格相结合，实现了 2D 和 3D 特征学习组合，从而达到了超过 13 的 mAP 改进。