Image synthesis has attracted emerging research interests in academic and
industry communities. Deep learning technologies especially the generative
models greatly inspired controllable image synthesis approaches and
applications, which aim to generate particular visual contents with latent
prompts. In order to further investigate low-level controllable image synthesis
problem which is crucial for fine image rendering and editing tasks, we present
a survey of some recent works on 3D controllable image synthesis using deep
learning. We first introduce the datasets and evaluation indicators for 3D
controllable image synthesis. Then, we review the state-of-the-art research for
geometrically controllable image synthesis in two aspects: 1)
Viewpoint/pose-controllable image synthesis; 2) Structure/shape-controllable
image synthesis. Furthermore, the photometrically controllable image synthesis
approaches are also reviewed for 3D re-lighting researches. While the emphasis
is on 3D controllable image synthesis algorithms, the related applications,
products and resources are also briefly summarized for practitioners.

通过深度学习技术与生成模型的应用，研究了 3D 可控图像合成问题，包括视角 / 姿势可控图像合成、结构 / 形状可控图像合成以及 3D 重新照明等方面。

深度学习下可控图像合成的调查

Survey on Controlable Image Synthesis with Deep Learning

In recent years, Generative Adversarial Networks have achieved impressive
results in photorealistic image synthesis. This progress nurtures hopes that
one day the classical rendering pipeline can be replaced by efficient models
that are learned directly from images. However, current image synthesis models
operate in the 2D domain where disentangling 3D properties such as camera
viewpoint or object pose is challenging. Furthermore, they lack an
interpretable and controllable representation. Our key hypothesis is that the
image generation process should be modeled in 3D space as the physical world
surrounding us is intrinsically three-dimensional. We define the new task of 3D
controllable image synthesis and propose an approach for solving it by
reasoning both in 3D space and in the 2D image domain. We demonstrate that our
model is able to disentangle latent 3D factors of simple multi-object scenes in
an unsupervised fashion from raw images. Compared to pure 2D baselines, it
allows for synthesizing scenes that are consistent wrt. changes in viewpoint or
object pose. We further evaluate various 3D representations in terms of their
usefulness for this challenging task.

该研究探索了使用生成对抗网络在三维空间中进行图像合成的可能性，并提出了一种无监督的方法，可从原始图像中解开简单场景的隐含三维因素。与纯二维基线相比，该方法允许合成与视角或物体姿势变化一致的场景，可用于实现 3D 可控图像合成，生成具有可解释性的表示形式。