3D-controllable portrait synthesis has significantly advanced, thanks to
breakthroughs in generative adversarial networks (GANs). However, it is still
challenging to manipulate existing face images with precise 3D control. While
concatenating GAN inversion and a 3D-aware, noise-to-image GAN is a
straight-forward solution, it is inefficient and may lead to noticeable drop in
editing quality. To fill this gap, we propose 3D-FM GAN, a novel conditional
GAN framework designed specifically for 3D-controllable face manipulation, and
does not require any tuning after the end-to-end learning phase. By carefully
encoding both the input face image and a physically-based rendering of 3D edits
into a StyleGAN's latent spaces, our image generator provides high-quality,
identity-preserved, 3D-controllable face manipulation. To effectively learn
such novel framework, we develop two essential training strategies and a novel
multiplicative co-modulation architecture that improves significantly upon
naive schemes. With extensive evaluations, we show that our method outperforms
the prior arts on various tasks, with better editability, stronger identity
preservation, and higher photo-realism. In addition, we demonstrate a better
generalizability of our design on large pose editing and out-of-domain images.

提出了一种新的 3D 可控脸部操作方法：3D-FM GAN，通过将输入脸部图像和 3D 编辑的真实渲染相结合，可以提供高质量、身份保留、3D 可控的脸部操作，相较于现有技术，具有更好的可编辑性、更强的身份保护和更高的照片逼真度。