Although significant progress has been made to audio-driven talking face
generation, existing methods either neglect facial emotion or cannot be applied
to arbitrary subjects. In this paper, we propose the Emotion-Aware Motion Model
(EAMM) to generate one-shot emotional talking faces by involving an emotion
source video. Specifically, we first propose an Audio2Facial-Dynamics module,
which renders talking faces from audio-driven unsupervised zero- and
first-order key-points motion. Then through exploring the motion model's
properties, we further propose an Implicit Emotion Displacement Learner to
represent emotion-related facial dynamics as linearly additive displacements to
the previously acquired motion representations. Comprehensive experiments
demonstrate that by incorporating the results from both modules, our method can
generate satisfactory talking face results on arbitrary subjects with realistic
emotion patterns.

本文提出 Emotion-Aware Motion Model (EAMM)，结合音频与表情视频生成单次具有真实感情模式的表情化人脸。使用无需监督的零级和一级关键点运动生成姿态，再将情感相关的面部动态表示为线性可加位移，并实现了令人满意的结果。

EAMM: 基于音频的情感感知运动模型实现一次性情感对话人脸

EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware  Motion Model

Despite previous success in generating audio-driven talking heads, most of
the previous studies focus on the correlation between speech content and the
mouth shape. Facial emotion, which is one of the most important features on
natural human faces, is always neglected in their methods. In this work, we
present Emotional Video Portraits (EVP), a system for synthesizing high-quality
video portraits with vivid emotional dynamics driven by audios. Specifically,
we propose the Cross-Reconstructed Emotion Disentanglement technique to
decompose speech into two decoupled spaces, i.e., a duration-independent
emotion space and a duration dependent content space. With the disentangled
features, dynamic 2D emotional facial landmarks can be deduced. Then we propose
the Target-Adaptive Face Synthesis technique to generate the final high-quality
video portraits, by bridging the gap between the deduced landmarks and the
natural head poses of target videos. Extensive experiments demonstrate the
effectiveness of our method both qualitatively and quantitatively.

本文提出一种称为情感视频肖像 (EVP) 的系统，通过 Cross-Reconstructed Emotion Disentanglement 技术来将语音分解为情感和内容空间并提取二维情感面部特征，然后通过 Target-Adaptive Face Synthesis 技术来生成高质量的情感动态视频人像，该方法在定性和定量实验中均表现出良好的效果。