Talking head synthesis, an advanced method for generating portrait videos
from a still image driven by specific content, has garnered widespread
attention in virtual reality, augmented reality and game production. Recently,
significant breakthroughs have been made with the introduction of novel models
such as the transformer and the diffusion model. Current methods can not only
generate new content but also edit the generated material. This survey
systematically reviews the technology, categorizing it into three pivotal
domains: portrait generation, driven mechanisms, and editing techniques. We
summarize milestone studies and critically analyze their innovations and
shortcomings within each domain. Additionally, we organize an extensive
collection of datasets and provide a thorough performance analysis of current
methodologies based on various evaluation metrics, aiming to furnish a clear
framework and robust data support for future research. Finally, we explore
application scenarios of talking head synthesis, illustrate them with specific
cases, and examine potential future directions.

头像合成的技术和应用进行了全面的调查和总结，涵盖了头像生成、驱动机制、编辑技术等方面，同时还提供了大量的数据集和绩效分析，以支持未来的研究和应用。

对话者合成的综合分类和分析：肖像生成、驱动机制和编辑的技术

A Comprehensive Taxonomy and Analysis of Talking Head Synthesis:  Techniques for Portrait Generation, Driving Mechanisms, and Editing

Recent advancement in personalized image generation have unveiled the
intriguing capability of pre-trained text-to-image models on learning identity
information from a collection of portrait images. However, existing solutions
can be vulnerable in producing truthful details, and usually suffer from
several defects such as (i) The generated face exhibit its own unique
characteristics, \ie facial shape and facial feature positioning may not
resemble key characteristics of the input, and (ii) The synthesized face may
contain warped, blurred or corrupted regions. In this paper, we present
FaceChain, a personalized portrait generation framework that combines a series
of customized image-generation model and a rich set of face-related perceptual
understanding models (\eg, face detection, deep face embedding extraction, and
facial attribute recognition), to tackle aforementioned challenges and to
generate truthful personalized portraits, with only a handful of portrait
images as input. Concretely, we inject several SOTA face models into the
generation procedure, achieving a more efficient label-tagging,
data-processing, and model post-processing compared to previous solutions, such
as DreamBooth ~\cite{ruiz2023dreambooth} , InstantBooth
~\cite{shi2023instantbooth} , or other LoRA-only approaches ~\cite{hu2021lora}
. Through the development of FaceChain, we have identified several potential
directions to accelerate development of Face/Human-Centric AIGC research and
application. We have designed FaceChain as a framework comprised of pluggable
components that can be easily adjusted to accommodate different styles and
personalized needs. We hope it can grow to serve the burgeoning needs from the
communities. FaceChain is open-sourced under Apache-2.0 license at
https://github.com/modelscope/facechain.

FaceChain 是一个个性化肖像生成框架，结合一系列定制图像生成模型和丰富的与人脸相关的感知理解模型，通过只有少量肖像图像作为输入，解决了现有解决方案中产生真实细节的漏洞和包含扭曲、模糊或损坏区域的合成人脸问题。

FaceChain：一个保留身份的肖像生成平台

FaceChain: A Playground for Identity-Preserving Portrait Generation

Over the years, 2D GANs have achieved great successes in photorealistic
portrait generation. However, they lack 3D understanding in the generation
process, thus they suffer from multi-view inconsistency problem. To alleviate
the issue, many 3D-aware GANs have been proposed and shown notable results, but
3D GANs struggle with editing semantic attributes. The controllability and
interpretability of 3D GANs have not been much explored. In this work, we
propose two solutions to overcome these weaknesses of 2D GANs and 3D-aware
GANs. We first introduce a novel 3D-aware GAN, SURF-GAN, which is capable of
discovering semantic attributes during training and controlling them in an
unsupervised manner. After that, we inject the prior of SURF-GAN into StyleGAN
to obtain a high-fidelity 3D-controllable generator. Unlike existing
latent-based methods allowing implicit pose control, the proposed
3D-controllable StyleGAN enables explicit pose control over portrait
generation. This distillation allows direct compatibility between 3D control
and many StyleGAN-based techniques (e.g., inversion and stylization), and also
brings an advantage in terms of computational resources. Our codes are
available at this https URL

通过引入 SURF-GAN 和将其先验注入 StyleGAN，提出了一种新的三维可控生成器，可进行人像生成的明确姿态控制。