We propose an audio-driven talking-head method to generate photo-realistic
talking-head videos from a single reference image. In this work, we tackle two
key challenges: (i) producing natural head motions that match speech prosody,
and (ii) maintaining the appearance of a speaker in a large head motion while
stabilizing the non-face regions. We first design a head pose predictor by
modeling rigid 6D head movements with a motion-aware recurrent neural network
(RNN). In this way, the predicted head poses act as the low-frequency holistic
movements of a talking head, thus allowing our latter network to focus on
detailed facial movement generation. To depict the entire image motions arising
from audio, we exploit a keypoint based dense motion field representation.
Then, we develop a motion field generator to produce the dense motion fields
from input audio, head poses, and a reference image. As this keypoint based
representation models the motions of facial regions, head, and backgrounds
integrally, our method can better constrain the spatial and temporal
consistency of the generated videos. Finally, an image generation network is
employed to render photo-realistic talking-head videos from the estimated
keypoint based motion fields and the input reference image. Extensive
experiments demonstrate that our method produces videos with plausible head
motions, synchronized facial expressions, and stable backgrounds and
outperforms the state-of-the-art.

本文提出一种基于音频驱动的头像动画方法，该方法同时解决了面部动画和头部运动的问题，并且采用了神经网络来预测和生成头部和面部的动画以及背景的动作。通过关键点密集的运动场表示，本方法产生的头像动画具有很好的空间和时间连续性，并且性能优于现有技术。