We propose a novel model for 3D semantic completion from a single depth
image, based on a single encoder and three separate generators used to
reconstruct different geometric and semantic representations of the original
and completed scene, all sharing the same latent space. To transfer information
between the geometric and semantic branches of the network, we introduce paths
between them concatenating features at corresponding network layers. Motivated
by the limited amount of training samples from real scenes, an interesting
attribute of our architecture is the capacity to supplement the existing
dataset by generating a new training dataset with high quality, realistic
scenes that even includes occlusion and real noise. We build the new dataset by
sampling the features directly from latent space which generates a pair of
partial volumetric surface and completed volumetric semantic surface. Moreover,
we utilize multiple discriminators to increase the accuracy and realism of the
reconstructions. We demonstrate the benefits of our approach on standard
benchmarks for the two most common completion tasks: semantic 3D scene
completion and 3D object completion.

提出了一种基于单个编码器和三个独立生成器的新型模型，用于从单个深度图像中进行三维语义完形填空，旨在实现对原始场景和完整场景的不同几何和语义表示的重构，在模型内引入连接它们之间的路径，在相应的网络层级上连接特征，以实现信息传输，并构建了一个新的训练数据集，包括遮挡和真实噪声，在标准基准上演示了我们的方法对语义三维场景和三维物体完成的好处。