Creating realistic or stylized facial and lip sync animation is a tedious
task. It requires lot of time and skills to sync the lips with audio and convey
the right emotion to the character's face. To allow animators to spend more
time on the artistic and creative part of the animation, we present Audio2Rig:
a new deep learning based tool leveraging previously animated sequences of a
show, to generate facial and lip sync rig animation from an audio file. Based
in Maya, it learns from any production rig without any adjustment and generates
high quality and stylized animations which mimic the style of the show.
Audio2Rig fits in the animator workflow: since it generates keys on the rig
controllers, the animation can be easily retaken. The method is based on 3
neural network modules which can learn an arbitrary number of controllers.
Hence, different configurations can be created for specific parts of the face
(such as the tongue, lips or eyes). With Audio2Rig, animators can also pick
different emotions and adjust their intensities to experiment or customize the
output, and have high level controls on the keyframes setting. Our method shows
excellent results, generating fine animation details while respecting the show
style. Finally, as the training relies on the studio data and is done
internally, it ensures data privacy and prevents from copyright infringement.

Audio2Rig 是一种基于深度学习的工具，通过从音频文件中生成面部和嘴唇同步的动画来减少动画师所需的时间和技能，同时保持高质量和风格一致性。

Audio2Rig：面向艺术家的面部动画深度学习工具

Audio2Rig: Artist-oriented deep learning tool for facial animation

To be widely adopted, 3D facial avatars need to be animated easily,
realistically, and directly, from speech signals. While the best recent methods
generate 3D animations that are synchronized with the input audio, they largely
ignore the impact of emotions on facial expressions. Instead, their focus is on
modeling the correlations between speech and facial motion, resulting in
animations that are unemotional or do not match the input emotion. We observe
that there are two contributing factors resulting in facial animation - the
speech and the emotion. We exploit these insights in EMOTE (Expressive Model
Optimized for Talking with Emotion), which generates 3D talking head avatars
that maintain lip sync while enabling explicit control over the expression of
emotion. Due to the absence of high-quality aligned emotional 3D face datasets
with speech, EMOTE is trained from an emotional video dataset (i.e., MEAD). To
achieve this, we match speech-content between generated sequences and target
videos differently from emotion content. Specifically, we train EMOTE with
additional supervision in the form of a lip-reading objective to preserve the
speech-dependent content (spatially local and high temporal frequency), while
utilizing emotion supervision on a sequence-level (spatially global and low
frequency). Furthermore, we employ a content-emotion exchange mechanism in
order to supervise different emotion on the same audio, while maintaining the
lip motion synchronized with the speech. To employ deep perceptual losses
without getting undesirable artifacts, we devise a motion prior in form of a
temporal VAE. Extensive qualitative, quantitative, and perceptual evaluations
demonstrate that EMOTE produces state-of-the-art speech-driven facial
animations, with lip sync on par with the best methods while offering
additional, high-quality emotional control.

为了实现面部三维动画的情感表达和语音的同步，研究者开发了一种名为 EMOTE 的系统，它通过在空间局部和时间高频率上对语音内容进行口型识别来训练表情，同时维持与语音同步且具有完全的情感表达。

情感演讲驱动的动画，包含内容 - 情感解缠技术

Emotional Speech-Driven Animation with Content-Emotion Disentanglement

The emergence of commercial tools for real-time performance-based 2D
animation has enabled 2D characters to appear on live broadcasts and streaming
platforms. A key requirement for live animation is fast and accurate lip sync
that allows characters to respond naturally to other actors or the audience
through the voice of a human performer. In this work, we present a deep
learning based interactive system that automatically generates live lip sync
for layered 2D characters using a Long Short Term Memory (LSTM) model. Our
system takes streaming audio as input and produces viseme sequences with less
than 200ms of latency (including processing time). Our contributions include
specific design decisions for our feature definition and LSTM configuration
that provide a small but useful amount of lookahead to produce accurate lip
sync. We also describe a data augmentation procedure that allows us to achieve
good results with a very small amount of hand-animated training data (13-20
minutes). Extensive human judgement experiments show that our results are
preferred over several competing methods, including those that only support
offline (non-live) processing. Video summary and supplementary results at
GitHub link: this https URL

本论文介绍了一种基于深度学习和 LSTM 模型的互动系统，该系统可以自动生成分层 2D 角色的实时唇形同步。采用小而有用的前瞻性设计，结合数据增强过程，可以在很少手工制作的培训数据下获得好的结果。

实时 2D 动画唇同步

Real-Time Lip Sync for Live 2D Animation

We present a novel approach to generating photo-realistic images of a face
with accurate lip sync, given an audio input. By using a recurrent neural
network, we achieved mouth landmarks based on audio features. We exploited the
power of conditional generative adversarial networks to produce
highly-realistic face conditioned on a set of landmarks. These two networks
together are capable of producing a sequence of natural faces in sync with an
input audio track.

利用递归神经网络和条件生成对抗网络，根据音频输入生成具有准确口型同步的逼真面部图像。