This paper presents a novel approach for text/speech-driven animation of a
photo-realistic head model based on blend-shape geometry, dynamic textures, and
neural rendering. Training a VAE for geometry and texture yields a parametric
model for accurate capturing and realistic synthesis of facial expressions from
a latent feature vector. Our animation method is based on a conditional CNN
that transforms text or speech into a sequence of animation parameters. In
contrast to previous approaches, our animation model learns
disentangling/synthesizing different acting-styles in an unsupervised manner,
requiring only phonetic labels that describe the content of training sequences.
For realistic real-time rendering, we train a U-Net that refines
rasterization-based renderings by computing improved pixel colors and a
foreground matte. We compare our framework qualitatively/quantitatively against
recent methods for head modeling as well as facial animation and evaluate the
perceived rendering/animation quality in a user-study, which indicates large
improvements compared to state-of-the-art approaches

本文提出了一种基于混合形状几何、动态纹理和神经渲染的照片级头部模型的文本 / 语音驱动动画的新方法，该动画方法基于有条件的 CNN 将文本或语音转换为一系列动画参数。