Collecting diverse sets of training images for RGB-D semantic image
segmentation is not always possible. In particular, when robots need to operate
in privacy-sensitive areas like homes, the collection is often limited to a
small set of locations. As a consequence, the annotated images lack diversity
in appearance and approaches for RGB-D semantic image segmentation tend to
overfit the training data. In this paper, we thus introduce semantic RGB-D
image synthesis to address this problem. It requires synthesising a
realistic-looking RGB-D image for a given semantic label map. Current
approaches, however, are uni-modal and cannot cope with multi-modal data.
Indeed, we show that extending uni-modal approaches to multi-modal data does
not perform well. In this paper, we therefore propose a generator for
multi-modal data that separates modal-independent information of the semantic
layout from the modal-dependent information that is needed to generate an RGB
and a depth image, respectively. Furthermore, we propose a discriminator that
ensures semantic consistency between the label maps and the generated images
and perceptual similarity between the real and generated images. Our
comprehensive experiments demonstrate that the proposed method outperforms
previous uni-modal methods by a large margin and that the accuracy of an
approach for RGB-D semantic segmentation can be significantly improved by
mixing real and generated images during training.

提出了一种用于 RGB-D 语义图像合成的多模态数据生成器，并通过鉴别器来确保标签图和生成图像之间的语义一致性及真实图像和生成图像之间的感知相似性，实验证明该方法在 RGB-D 语义分割方案中表现出了显著优势，以及通过在训练过程中混合真实图像和生成图像可以显著提高方案的准确性。