A key challenge for novel view synthesis of monocular portrait images is 3D
consistency under continuous pose variations. Most existing methods rely on 2D
generative models which often leads to obvious 3D inconsistency artifacts. We
present a 3D-consistent novel view synthesis approach for monocular portrait
images based on a recent proposed 3D-aware GAN, namely Generative Radiance
Manifolds (GRAM), which has shown strong 3D consistency at multiview image
generation of virtual subjects via the radiance manifolds representation.
However, simply learning an encoder to map a real image into the latent space
of GRAM can only reconstruct coarse radiance manifolds without faithful fine
details, while improving the reconstruction fidelity via instance-specific
optimization is time-consuming. We introduce a novel detail manifolds
reconstructor to learn 3D-consistent fine details on the radiance manifolds
from monocular images, and combine them with the coarse radiance manifolds for
high-fidelity reconstruction. The 3D priors derived from the coarse radiance
manifolds are used to regulate the learned details to ensure reasonable
synthesized results at novel views. Trained on in-the-wild 2D images, our
method achieves high-fidelity and 3D-consistent portrait synthesis largely
outperforming the prior art.

该研究基于生成光辐射流流形模型的最近提出的 3D 感知 GAN-Gram，介绍了一种适用于单目头像的三维相容视角合成方法，该方法使用细节流形重构器来学习纹理细节，并将其与光辐射流流形相结合，以实现高保真三维头像重建。