Digital humans and, especially, 3D facial avatars have raised a lot of
attention in the past years, as they are the backbone of several applications
like immersive telepresence in AR or VR. Despite the progress, facial avatars
reconstructed from commodity hardware are incomplete and miss out on parts of
the side and back of the head, severely limiting the usability of the avatar.
This limitation in prior work stems from their requirement of face tracking,
which fails for profile and back views. To address this issue, we propose to
learn person-specific animatable avatars from images without assuming to have
access to precise facial expression tracking. At the core of our method, we
leverage a 3D-aware generative model that is trained to reproduce the
distribution of facial expressions from the training data. To train this
appearance model, we only assume to have a collection of 2D images with the
corresponding camera parameters. For controlling the model, we learn a mapping
from 3DMM facial expression parameters to the latent space of the generative
model. This mapping can be learned by sampling the latent space of the
appearance model and reconstructing the facial parameters from a normalized
frontal view, where facial expression estimation performs well. With this
scheme, we decouple 3D appearance reconstruction and animation control to
achieve high fidelity in image synthesis. In a series of experiments, we
compare our proposed technique to state-of-the-art monocular methods and show
superior quality while not requiring expression tracking of the training data.

我们提出了一种从图像中学习具有人物特定的可动画化角色模型的方法，旨在解决面部表情追踪失败的问题，并实现高保真度的图像合成。

GAN-Avatar: 可控个性化基于 GAN 的人头图像生成器

GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar

To be widely adopted, 3D facial avatars need to be animated easily,
realistically, and directly, from speech signals. While the best recent methods
generate 3D animations that are synchronized with the input audio, they largely
ignore the impact of emotions on facial expressions. Instead, their focus is on
modeling the correlations between speech and facial motion, resulting in
animations that are unemotional or do not match the input emotion. We observe
that there are two contributing factors resulting in facial animation - the
speech and the emotion. We exploit these insights in EMOTE (Expressive Model
Optimized for Talking with Emotion), which generates 3D talking head avatars
that maintain lip sync while enabling explicit control over the expression of
emotion. Due to the absence of high-quality aligned emotional 3D face datasets
with speech, EMOTE is trained from an emotional video dataset (i.e., MEAD). To
achieve this, we match speech-content between generated sequences and target
videos differently from emotion content. Specifically, we train EMOTE with
additional supervision in the form of a lip-reading objective to preserve the
speech-dependent content (spatially local and high temporal frequency), while
utilizing emotion supervision on a sequence-level (spatially global and low
frequency). Furthermore, we employ a content-emotion exchange mechanism in
order to supervise different emotion on the same audio, while maintaining the
lip motion synchronized with the speech. To employ deep perceptual losses
without getting undesirable artifacts, we devise a motion prior in form of a
temporal VAE. Extensive qualitative, quantitative, and perceptual evaluations
demonstrate that EMOTE produces state-of-the-art speech-driven facial
animations, with lip sync on par with the best methods while offering
additional, high-quality emotional control.

为了实现面部三维动画的情感表达和语音的同步，研究者开发了一种名为 EMOTE 的系统，它通过在空间局部和时间高频率上对语音内容进行口型识别来训练表情，同时维持与语音同步且具有完全的情感表达。

情感演讲驱动的动画，包含内容 - 情感解缠技术

Emotional Speech-Driven Animation with Content-Emotion Disentanglement

As 3D facial avatars become more widely used for communication, it is
critical that they faithfully convey emotion. Unfortunately, the best recent
methods that regress parametric 3D face models from monocular images are unable
to capture the full spectrum of facial expression, such as subtle or extreme
emotions. We find the standard reconstruction metrics used for training
(landmark reprojection error, photometric error, and face recognition loss) are
insufficient to capture high-fidelity expressions. The result is facial
geometries that do not match the emotional content of the input image. We
address this with EMOCA (EMOtion Capture and Animation), by introducing a novel
deep perceptual emotion consistency loss during training, which helps ensure
that the reconstructed 3D expression matches the expression depicted in the
input image. While EMOCA achieves 3D reconstruction errors that are on par with
the current best methods, it significantly outperforms them in terms of the
quality of the reconstructed expression and the perceived emotional content. We
also directly regress levels of valence and arousal and classify basic
expressions from the estimated 3D face parameters. On the task of in-the-wild
emotion recognition, our purely geometric approach is on par with the best
image-based methods, highlighting the value of 3D geometry in analyzing human
behavior. The model and code are publicly available at
this https URL

研究使用 3D 面部化身进行情感传达，提出了一种新的深度知觉情感一致性丢失训练方法 EMOCA，实现了高保真的面部表情重建和情感传达，可用于分析人类行为。