This paper tackles the problem of data fusion in the semantic scene
completion (SSC) task, which can simultaneously deal with semantic labeling and
scene completion. RGB images contain texture details of the object(s) which are
vital for semantic scene understanding. Meanwhile, depth images capture
geometric clues of high relevance for shape completion. Using both RGB and
depth images can further boost the accuracy of SSC over employing one modality
in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which
learns to adaptively select and fuse the relevant information from depth and
RGB by making use of the gate and memory modules. Based on the single-stage
fusion, we further propose a multi-stage fusion strategy, which could model the
correlations among different stages within the network. Extensive experiments
on two benchmark datasets demonstrate the superior performance and the
effectiveness of the proposed GRFNet for data fusion in SSC. Code will be made
available.

本文针对语义场景完成（SSC）任务中的数据融合问题进行了研究，我们提出了一种 3D 门控递归融合网络（GRFNet），其可以自适应地选择和融合来自深度和 RGB 图像的相关信息，并借助门控和记忆模块。基于单阶段融合，我们进一步提出了一种多阶段融合策略，并在两个基准数据集上进行了广泛的实验，证明了所提出的 GRFNet 在 SSC 中的数据融合方面具有卓越的性能和有效性。