We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D
scans in indoor environments using a joint 3D-multi-view prediction network. In
contrast to existing methods that either use geometry or RGB data as input for
this task, we combine both data modalities in a joint, end-to-end network
architecture. Rather than simply projecting color data into a volumetric grid
and operating solely in 3D -- which would result in insufficient detail -- we
first extract feature maps from associated RGB images. These features are then
mapped into the volumetric feature grid of a 3D network using a differentiable
backprojection layer. Since our target is 3D scanning scenarios with possibly
many frames, we use a multi-view pooling approach in order to handle a varying
number of RGB input views. This learned combination of RGB and geometric
features with our joint 2D-3D architecture achieves significantly better
results than existing baselines. For instance, our final result on the ScanNet
3D segmentation benchmark increases from 52.8\% to 75\% accuracy compared to
existing volumetric architectures.

提出了一种名为 3DMV 的新型方法，使用联合 3D - 多视图预测网络对室内环境中 RGB-D 扫描进行三维语义场景分割。通过将 RGB 和几何特征相结合，并以一个可微的 backprojection 层将其映射到三维网格中，这种联合二维 - 三维架构实现了比现有基线方法显著更好的结果。