To be widely adopted, 3D facial avatars need to be animated easily,
realistically, and directly, from speech signals. While the best recent methods
generate 3D animations that are synchronized with the input audio, they largely
ignore the impact of emotions on facial expressions. Instead, their focus is on
modeling the correlations between speech and facial motion, resulting in
animations that are unemotional or do not match the input emotion. We observe
that there are two contributing factors resulting in facial animation - the
speech and the emotion. We exploit these insights in EMOTE (Expressive Model
Optimized for Talking with Emotion), which generates 3D talking head avatars
that maintain lip sync while enabling explicit control over the expression of
emotion. Due to the absence of high-quality aligned emotional 3D face datasets
with speech, EMOTE is trained from an emotional video dataset (i.e., MEAD). To
achieve this, we match speech-content between generated sequences and target
videos differently from emotion content. Specifically, we train EMOTE with
additional supervision in the form of a lip-reading objective to preserve the
speech-dependent content (spatially local and high temporal frequency), while
utilizing emotion supervision on a sequence-level (spatially global and low
frequency). Furthermore, we employ a content-emotion exchange mechanism in
order to supervise different emotion on the same audio, while maintaining the
lip motion synchronized with the speech. To employ deep perceptual losses
without getting undesirable artifacts, we devise a motion prior in form of a
temporal VAE. Extensive qualitative, quantitative, and perceptual evaluations
demonstrate that EMOTE produces state-of-the-art speech-driven facial
animations, with lip sync on par with the best methods while offering
additional, high-quality emotional control.

为了实现面部三维动画的情感表达和语音的同步，研究者开发了一种名为 EMOTE 的系统，它通过在空间局部和时间高频率上对语音内容进行口型识别来训练表情，同时维持与语音同步且具有完全的情感表达。