We introduce a deep appearance model for rendering the human face. Inspired
by Active Appearance Models, we develop a data-driven rendering pipeline that
learns a joint representation of facial geometry and appearance from a
multiview capture setup. Vertex positions and view-specific textures are
modeled using a deep variational autoencoder that captures complex nonlinear
effects while producing a smooth and compact latent representation.
View-specific texture enables the modeling of view-dependent effects such as
specularity. In addition, it can also correct for imperfect geometry stemming
from biased or low resolution estimates. This is a significant departure from
the traditional graphics pipeline, which requires highly accurate geometry as
well as all elements of the shading model to achieve realism through
physically-inspired light transport. Acquiring such a high level of accuracy is
difficult in practice, especially for complex and intricate parts of the face,
such as eyelashes and the oral cavity. These are handled naturally by our
approach, which does not rely on precise estimates of geometry. Instead, the
shading model accommodates deficiencies in geometry though the flexibility
afforded by the neural network employed. At inference time, we condition the
decoding network on the viewpoint of the camera in order to generate the
appropriate texture for rendering. The resulting system can be implemented
simply using existing rendering engines through dynamic textures with flat
lighting. This representation, together with a novel unsupervised technique for
mapping images to facial states, results in a system that is naturally suited
to real-time interactive settings such as Virtual Reality (VR).

本文介绍了一种基于多视角捕捉技术，利用深度变分自编码器来学习人脸的几何结构和外观特征联合表示，可实现对大规模、高度复杂的人脸几何结构的完美表现，从而能够适用于实时互动场景，例如虚拟现实 (VR) 应用。