Current generative frameworks use end-to-end learning and generate images by
sampling from uniform noise distribution. However, these approaches ignore the
most basic principle of image formation: images are product of: (a) Structure:
the underlying 3D model; (b) Style: the texture mapped onto structure. In this
paper, we factorize the image generation process and propose Style and
Structure Generative Adversarial Network (S^2-GAN). Our S^2-GAN has two
components: the Structure-GAN generates a surface normal map; the Style-GAN
takes the surface normal map as input and generates the 2D image. Apart from a
real vs. generated loss function, we use an additional loss with computed
surface normals from generated images. The two GANs are first trained
independently, and then merged together via joint learning. We show our S^2-GAN
model is interpretable, generates more realistic images and can be used to
learn unsupervised RGBD representations.

本研究提出了一个解决图像生成的新框架，通过拆分成结构生成和风格生成两个 GAN 组件，使用附加的损失函数和联合学习，该模型可以生成更真实的图像，用于学习无监督的 RGBD 表示。