In recent years, Generative Adversarial Networks have achieved impressive
results in photorealistic image synthesis. This progress nurtures hopes that
one day the classical rendering pipeline can be replaced by efficient models
that are learned directly from images. However, current image synthesis models
operate in the 2D domain where disentangling 3D properties such as camera
viewpoint or object pose is challenging. Furthermore, they lack an
interpretable and controllable representation. Our key hypothesis is that the
image generation process should be modeled in 3D space as the physical world
surrounding us is intrinsically three-dimensional. We define the new task of 3D
controllable image synthesis and propose an approach for solving it by
reasoning both in 3D space and in the 2D image domain. We demonstrate that our
model is able to disentangle latent 3D factors of simple multi-object scenes in
an unsupervised fashion from raw images. Compared to pure 2D baselines, it
allows for synthesizing scenes that are consistent wrt. changes in viewpoint or
object pose. We further evaluate various 3D representations in terms of their
usefulness for this challenging task.

该研究探索了使用生成对抗网络在三维空间中进行图像合成的可能性，并提出了一种无监督的方法，可从原始图像中解开简单场景的隐含三维因素。与纯二维基线相比，该方法允许合成与视角或物体姿势变化一致的场景，可用于实现 3D 可控图像合成，生成具有可解释性的表示形式。