Current Generative Adversarial Networks (GANs) produce photorealistic
renderings of portrait images. Embedding real images into the latent space of
such models enables high-level image editing. While recent methods provide
considerable semantic control over the (re-)generated images, they can only
generate a limited set of viewpoints and cannot explicitly control the camera.
Such 3D camera control is required for 3D virtual and mixed reality
applications. In our solution, we use a few images of a face to perform 3D
reconstruction, and we introduce the notion of the GAN camera manifold, the key
element allowing us to precisely define the range of images that the GAN can
reproduce in a stable manner. We train a small face-specific neural implicit
representation network to map a captured face to this manifold and complement
it with a warping scheme to obtain free-viewpoint novel-view synthesis. We show
how our approach - due to its precise camera control - enables the integration
of a pre-trained StyleGAN into standard 3D rendering pipelines, allowing e.g.,
stereo rendering or consistent insertion of faces in synthetic 3D environments.
Our solution proposes the first truly free-viewpoint rendering of realistic
faces at interactive rates, using only a small number of casual photos as
input, while simultaneously allowing semantic editing capabilities, such as
facial expression or lighting changes.

本文提出了利用 GAN 进行 3D 重建，并引入了 GAN 相机流形的概念，通过精准的相机控制，实现了真正自由视点的渲染，在输入少量的照片的同时具有明显的语义控制能力。