Learning 3D generative models from a dataset of monocular images enables
self-supervised 3D reasoning and controllable synthesis. State-of-the-art 3D
generative models are GANs which use neural 3D volumetric representations for
synthesis. Images are synthesized by rendering the volumes from a given camera.
These models can disentangle the 3D scene from the camera viewpoint in any
generated image. However, most models do not disentangle other factors of image
formation, such as geometry and appearance. In this paper, we design a 3D GAN
which can learn a disentangled model of objects, just from monocular
observations. Our model can disentangle the geometry and appearance variations
in the scene, i.e., we can independently sample from the geometry and
appearance spaces of the generative model. This is achieved using a novel
non-rigid deformable scene formulation. A 3D volume which represents an object
instance is computed as a non-rigidly deformed canonical 3D volume. Our method
learns the canonical volume, as well as its deformations, jointly during
training. This formulation also helps us improve the disentanglement between
the 3D scene and the camera viewpoints using a novel pose regularization loss
defined on the 3D deformation field. In addition, we further model the inverse
deformations, enabling the computation of dense correspondences between images
generated by our model. Finally, we design an approach to embed real images
into the latent space of our disentangled generative model, enabling editing of
real images.

本文提出了一种基于 3D 生成模型的非刚性可变形场景表达方法，通过联合学习规范模型及其变形，并使用姿态正则化损失来改善模型的场景与摄影视角的分离性，同时还可以嵌入真实图像中并进行编辑。