Deep generative models can synthesize photorealistic images of human faces
with novel identities. However, a key challenge to the wide applicability of
such techniques is to provide independent control over semantically meaningful
parameters: appearance, head pose, face shape, and facial expressions. In this
paper, we propose VariTex - to the best of our knowledge the first method that
learns a variational latent feature space of neural face textures, which allows
sampling of novel identities. We combine this generative model with a
parametric face model and gain explicit control over head pose and facial
expressions. To generate complete images of human heads, we propose an additive
decoder that adds plausible details such as hair. A novel training scheme
enforces a pose-independent latent space and in consequence, allows learning a
one-to-many mapping between latent codes and pose-conditioned exterior regions.
The resulting method can generate geometrically consistent images of novel
identities under fine-grained control over head pose, face shape, and facial
expressions. This facilitates a broad range of downstream tasks, like sampling
novel identities, changing the head pose, expression transfer, and more. Code
and models are available for research on this https URL

该研究提出了 VariTex 方法，利用神经人脸纹理学习了变分潜在特征空间，使得其能够控制面部外部区域，并以此实现对外貌、头部姿态和面部表情的精细控制，从而能够合成具有几何一致性的人脸图片。

VariTex: 变分神经面部纹理

VariTex: Variational Neural Face Textures

Accurate modeling of 3D objects exhibiting transparency, reflections and thin
structures is an extremely challenging problem. Inspired by billboards and
geometric proxies used in computer graphics, this paper proposes Generative
Latent Textured Objects (GeLaTO), a compact representation that combines a set
of coarse shape proxies defining low frequency geometry with learned neural
textures, to encode both medium and fine scale geometry as well as
view-dependent appearance. To generate the proxies' textures, we learn a joint
latent space allowing category-level appearance and geometry interpolation. The
proxies are independently rasterized with their corresponding neural texture
and composited using a U-Net, which generates an output photorealistic image
including an alpha map. We demonstrate the effectiveness of our approach by
reconstructing complex objects from a sparse set of views. We show results on a
dataset of real images of eyeglasses frames, which are particularly challenging
to reconstruct using classical methods. We also demonstrate that these coarse
proxies can be handcrafted when the underlying object geometry is easy to
model, like eyeglasses, or generated using a neural network for more complex
categories, such as cars.

该研究提出了一种名为 GeLaTO 的紧凑表示法，它结合了描述低频几何的一组粗略形状代理和学习的神经纹理，以编码介质和细尺度几何以及视角相关的外观，并应用 U-Net 将代理分别光栅化与对应的神经纹理进行组合，生成具有逼真外观的图像，该方法可用于复杂物体的重建。

GeLaTO: 生成潜在的纹理物体

GeLaTO: Generative Latent Textured Objects

The modern computer graphics pipeline can synthesize images at remarkable
visual quality; however, it requires well-defined, high-quality 3D content as
input. In this work, we explore the use of imperfect 3D content, for instance,
obtained from photo-metric reconstructions with noisy and incomplete surface
geometry, while still aiming to produce photo-realistic (re-)renderings. To
address this challenging problem, we introduce Deferred Neural Rendering, a new
paradigm for image synthesis that combines the traditional graphics pipeline
with learnable components. Specifically, we propose Neural Textures, which are
learned feature maps that are trained as part of the scene capture process.
Similar to traditional textures, neural textures are stored as maps on top of
3D mesh proxies; however, the high-dimensional feature maps contain
significantly more information, which can be interpreted by our new deferred
neural rendering pipeline. Both neural textures and deferred neural renderer
are trained end-to-end, enabling us to synthesize photo-realistic images even
when the original 3D content was imperfect. In contrast to traditional,
black-box 2D generative neural networks, our 3D representation gives us
explicit control over the generated output, and allows for a wide range of
application domains. For instance, we can synthesize temporally-consistent
video re-renderings of recorded 3D scenes as our representation is inherently
embedded in 3D space. This way, neural textures can be utilized to coherently
re-render or manipulate existing video content in both static and dynamic
environments at real-time rates. We show the effectiveness of our approach in
several experiments on novel view synthesis, scene editing, and facial
reenactment, and compare to state-of-the-art approaches that leverage the
standard graphics pipeline as well as conventional generative neural networks.

通过引入 Deferred Neural Rendering 和 Neural Textures，本文提出一种新的渲染模型，可以快速准确地合成出高质量的视觉效果，即使输入的 3D 内容不完整嘈杂，同时可以在多个应用领域内进行使用，例如在新颖的视角合成，场景编辑和面部复制等方面。