Talking head synthesis is an emerging technology with wide applications in
film dubbing, virtual avatars and online education. Recent NeRF-based methods
generate more natural talking videos, as they better capture the 3D structural
information of faces. However, a specific model needs to be trained for each
identity with a large dataset. In this paper, we propose Dynamic Facial
Radiance Fields (DFRF) for few-shot talking head synthesis, which can rapidly
generalize to an unseen identity with few training data. Different from the
existing NeRF-based methods which directly encode the 3D geometry and
appearance of a specific person into the network, our DFRF conditions face
radiance field on 2D appearance images to learn the face prior. Thus the facial
radiance field can be flexibly adjusted to the new identity with few reference
images. Additionally, for better modeling of the facial deformations, we
propose a differentiable face warping module conditioned on audio signals to
deform all reference images to the query space. Extensive experiments show that
with only tens of seconds of training clip available, our proposed DFRF can
synthesize natural and high-quality audio-driven talking head videos for novel
identities with only 40k iterations. We highly recommend readers view our
supplementary video for intuitive comparisons. Code is available in
this https URL

本篇论文中，我们提出了一种名为 DFRT 的方法用于 few-shot 声音驱动下的面部融合。我们的方法使用 2D 图像来学习面部的先验知识，并且可以通过少量的训练数据应用于新的身份。同时，我们还提出了一种可微的面部扭曲模块以更好地建模面部变形。最终的实验结果表明，我们的方法可以在很短的时间内生成高质量的自然音频驱动的视频。