We investigate how to generate multimodal image outputs, such as RGB, depth,
and surface normals, with a single generative model. The challenge is to
produce outputs that are realistic, and also consistent with each other. Our
solution builds on the StyleGAN3 architecture, with a shared backbone and
modality-specific branches in the last layers of the synthesis network, and we
propose per-modality fidelity discriminators and a cross-modality consistency
discriminator. In experiments on the Stanford2D3D dataset, we demonstrate
realistic and consistent generation of RGB, depth, and normal images. We also
show a training recipe to easily extend our pretrained model on a new domain,
even with a few pairwise data. We further evaluate the use of synthetically
generated RGB and depth pairs for training or fine-tuning depth estimators.
Code will be available at this https URL

研究如何使用单个生成模型生成多模式图像输出，并使用 StyleGAN3 架构，通过获取每种模式的质量鉴别器和跨模式一致性鉴别器来解决图像一致性问题，并在 Stanford2D3D 数据集上展示了实际和一致的 RGB、深度和法线图像的生成。