This paper presents an end-to-end 3D convolutional network named
attention-based multi-modal fusion network (AMFNet) for the semantic scene
completion (SSC) task of inferring the occupancy and semantic labels of a
volumetric 3D scene from single-view RGB-D images. Compared with previous
methods which use only the semantic features extracted from RGB-D images, the
proposed AMFNet learns to perform effective 3D scene completion and semantic
segmentation simultaneously via leveraging the experience of inferring 2D
semantic segmentation from RGB-D images as well as the reliable depth cues in
spatial dimension. It is achieved by employing a multi-modal fusion
architecture boosted from 2D semantic segmentation and a 3D semantic completion
network empowered by residual attention blocks. We validate our method on both
the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset and the results
show that our method respectively achieves the gains of 2.5% and 2.6% on the
synthetic SUNCG-RGBD dataset and the real NYUv2 dataset against the
state-of-the-art method.

本文提出了一种名为 AMFNet 的端到端三维卷积网络，用于从单视图 RGB-D 图像推断出体积三维场景的占用和语义标签。该网络通过融合 2D 语义分割和 3D 语义完成网络的经验以及空间维度的可靠深度线索，学习同时执行有效的 3D 场景完成和语义分割，验证表明在合成 SUNCG-RGBD 数据集和真实 NYUv2 数据集上都取得了明显的优越性。