Speech-driven facial animation is the process that automatically synthesizes
talking characters based on speech signals. The majority of work in this domain
creates a mapping from audio features to visual features. This approach often
requires post-processing using computer graphics techniques to produce
realistic albeit subject dependent results. We present an end-to-end system
that generates videos of a talking head, using only a still image of a person
and an audio clip containing speech, without relying on handcrafted
intermediate features. Our method generates videos which have (a) lip movements
that are in sync with the audio and (b) natural facial expressions such as
blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on
achieving detailed frames, audio-visual synchronization, and realistic
expressions. We quantify the contribution of each component in our model using
an ablation study and we provide insights into the latent representation of the
model. The generated videos are evaluated based on sharpness, reconstruction
quality, lip-reading accuracy, synchronization as well as their ability to
generate natural blinks.

本文介绍了一种基于语音信号自动生成会说话的角色的方法，该方法使用了端到端的系统并使用 Generative Adversarial Network (生成对抗网络) 来确保视频与音频的实时同步，并且产生了自然的面部表情。