Generating talking head videos through a face image and a piece of speech
audio still contains many challenges. ie, unnatural head movement, distorted
expression, and identity modification. We argue that these issues are mainly
because of learning from the coupled 2D motion fields. On the other hand,
explicitly using 3D information also suffers problems of stiff expression and
incoherent video. We present SadTalker, which generates 3D motion coefficients
(head pose, expression) of the 3DMM from audio and implicitly modulates a novel
3D-aware face render for talking head generation. To learn the realistic motion
coefficients, we explicitly model the connections between audio and different
types of motion coefficients individually. Precisely, we present ExpNet to
learn the accurate facial expression from audio by distilling both coefficients
and 3D-rendered faces. As for the head pose, we design PoseVAE via a
conditional VAE to synthesize head motion in different styles. Finally, the
generated 3D motion coefficients are mapped to the unsupervised 3D keypoints
space of the proposed face render, and synthesize the final video. We conducted
extensive experiments to demonstrate the superiority of our method in terms of
motion and video quality.

本文提出了 SadTalker—— 一种生成逼真说话头像视频的方法，通过音频和 3D 运动系数生成 3D 头部姿态和表情，利用 ExpNet 和 PoseVAE 分别进行表情和头部姿态合成，并使用生成的 3D 运动系数合成最终视频，实验结果表明了该方法优于其他方法。