Talking head synthesis, an advanced method for generating portrait videos
from a still image driven by specific content, has garnered widespread
attention in virtual reality, augmented reality and game production. Recently,
significant breakthroughs have been made with the introduction of novel models
such as the transformer and the diffusion model. Current methods can not only
generate new content but also edit the generated material. This survey
systematically reviews the technology, categorizing it into three pivotal
domains: portrait generation, driven mechanisms, and editing techniques. We
summarize milestone studies and critically analyze their innovations and
shortcomings within each domain. Additionally, we organize an extensive
collection of datasets and provide a thorough performance analysis of current
methodologies based on various evaluation metrics, aiming to furnish a clear
framework and robust data support for future research. Finally, we explore
application scenarios of talking head synthesis, illustrate them with specific
cases, and examine potential future directions.

头像合成的技术和应用进行了全面的调查和总结，涵盖了头像生成、驱动机制、编辑技术等方面，同时还提供了大量的数据集和绩效分析，以支持未来的研究和应用。

对话者合成的综合分类和分析：肖像生成、驱动机制和编辑的技术

A Comprehensive Taxonomy and Analysis of Talking Head Synthesis:  Techniques for Portrait Generation, Driving Mechanisms, and Editing

We propose a novel talking head synthesis pipeline called "DiT-Head", which
is based on diffusion transformers and uses audio as a condition to drive the
denoising process of a diffusion model. Our method is scalable and can
generalise to multiple identities while producing high-quality results. We
train and evaluate our proposed approach and compare it against existing
methods of talking head synthesis. We show that our model can compete with
these methods in terms of visual quality and lip-sync accuracy. Our results
highlight the potential of our proposed approach to be used for a wide range of
applications, including virtual assistants, entertainment, and education. For a
video demonstration of the results and our user study, please refer to our
supplementary material.

我们提出了一种基于扩散变压器的新型对话头合成流程，利用音频作为条件来驱动扩散模型的去噪过程。我们的方法具有可扩展性，可以推广到多个身份，同时产生高质量的结果。通过与现有的对话头合成方法进行比较，我们对我们提出的方法进行训练和评估，并展示了我们的模型在视觉质量和嘴唇同步准确性方面可以与这些方法竞争。我们的结果突显了我们提出的方法在包括虚拟助手、娱乐和教育在内的广泛应用中的潜力。请参阅我们的补充材料以获取结果和用户研究的视频演示。

DiT-Head：使用扩散变压器进行高分辨率说话人合成

DiT-Head: High-Resolution Talking Head Synthesis using Diffusion  Transformers

Talking head synthesis is an emerging technology with wide applications in
film dubbing, virtual avatars and online education. Recent NeRF-based methods
generate more natural talking videos, as they better capture the 3D structural
information of faces. However, a specific model needs to be trained for each
identity with a large dataset. In this paper, we propose Dynamic Facial
Radiance Fields (DFRF) for few-shot talking head synthesis, which can rapidly
generalize to an unseen identity with few training data. Different from the
existing NeRF-based methods which directly encode the 3D geometry and
appearance of a specific person into the network, our DFRF conditions face
radiance field on 2D appearance images to learn the face prior. Thus the facial
radiance field can be flexibly adjusted to the new identity with few reference
images. Additionally, for better modeling of the facial deformations, we
propose a differentiable face warping module conditioned on audio signals to
deform all reference images to the query space. Extensive experiments show that
with only tens of seconds of training clip available, our proposed DFRF can
synthesize natural and high-quality audio-driven talking head videos for novel
identities with only 40k iterations. We highly recommend readers view our
supplementary video for intuitive comparisons. Code is available in
this https URL

本篇论文中，我们提出了一种名为 DFRT 的方法用于 few-shot 声音驱动下的面部融合。我们的方法使用 2D 图像来学习面部的先验知识，并且可以通过少量的训练数据应用于新的身份。同时，我们还提出了一种可微的面部扭曲模块以更好地建模面部变形。最终的实验结果表明，我们的方法可以在很短的时间内生成高质量的自然音频驱动的视频。