In this paper, we propose a novel text-based talking-head video generation
framework that synthesizes high-fidelity facial expressions and head motions in
accordance with contextual sentiments as well as speech rhythm and pauses. To
be specific, our framework consists of a speaker-independent stage and a
speaker-specific stage. In the speaker-independent stage, we design three
parallel networks to generate animation parameters of the mouth, upper face,
and head from texts, separately. In the speaker-specific stage, we present a 3D
face model guided attention network to synthesize videos tailored for different
individuals. It takes the animation parameters as input and exploits an
attention mask to manipulate facial expression changes for the input
individuals. Furthermore, to better establish authentic correspondences between
visual motions (i.e., facial expression changes and head movements) and audios,
we leverage a high-accuracy motion capture dataset instead of relying on long
videos of specific individuals. After attaining the visual and audio
correspondences, we can effectively train our network in an end-to-end fashion.
Extensive experiments on qualitative and quantitative results demonstrate that
our algorithm achieves high-quality photo-realistic talking-head videos
including various facial expressions and head motions according to speech
rhythms and outperforms the state-of-the-art.

本研究提出了一种基于文本的交谈头视频生成框架，能够按照上下文情感及语音节奏和停顿，合成高保真度的面部表情和头部动作。我们的算法通过两个阶段：一个是针对多个人种实现的通用阶段，一个是基于每一个人的具体情况实现的个性阶段。通过广泛的实验验证，我们的这一算法能够生成高质量、逼真的交谈头视频，且在多项指标上均超过目前领先的技术水平。