When people deliver a speech, they naturally move heads, and this rhythmic
head motion conveys prosodic information. However, generating a lip-synced
video while moving head naturally is challenging. While remarkably successful,
existing works either generate still talkingface videos or rely on
landmark/video frames as sparse/dense mapping guidance to generate head
movements, which leads to unrealistic or uncontrollable video synthesis. To
overcome the limitations, we propose a 3D-aware generative network along with a
hybrid embedding module and a non-linear composition module. Through modeling
the head motion and facial expressions1 explicitly, manipulating 3D animation
carefully, and embedding reference images dynamically, our approach achieves
controllable, photo-realistic, and temporally coherent talking-head videos with
natural head movements. Thoughtful experiments on several standard benchmarks
demonstrate that our method achieves significantly better results than the
state-of-the-art methods in both quantitative and qualitative comparisons. The
code is available on this https URL
lelechen63/Talking-head-Generation-with-Rhythmic-Head-Motion.

本文提出一种基于 3D-aware 生成网络、混合嵌入和非线性合成模块的方法，通过显式建模头部运动和面部表情，精心处理 3D 动画以及动态嵌入参考图像，实现了可控、逼真、时序连贯的说话者头像视频，并在多个标准基准测试中表现出优异的结果。