Radiance fields have demonstrated impressive performance in synthesizing
lifelike 3D talking heads. However, due to the difficulty in fitting steep
appearance changes, the prevailing paradigm that presents facial motions by
directly modifying point appearance may lead to distortions in dynamic regions.
To tackle this challenge, we introduce TalkingGaussian, a deformation-based
radiance fields framework for high-fidelity talking head synthesis. Leveraging
the point-based Gaussian Splatting, facial motions can be represented in our
method by applying smooth and continuous deformations to persistent Gaussian
primitives, without requiring to learn the difficult appearance change like
previous methods. Due to this simplification, precise facial motions can be
synthesized while keeping a highly intact facial feature. Under such a
deformation paradigm, we further identify a face-mouth motion inconsistency
that would affect the learning of detailed speaking motions. To address this
conflict, we decompose the model into two branches separately for the face and
inside mouth areas, therefore simplifying the learning tasks to help
reconstruct more accurate motion and structure of the mouth region. Extensive
experiments demonstrate that our method renders high-quality lip-synchronized
talking head videos, with better facial fidelity and higher efficiency compared
with previous methods.

本研究提出了一种基于变形的辐射场框架 TalkingGaussian，用于高保真度语音同步头像合成，并通过将模型分为两个分支来解决面部和口内区域的运动不一致问题，从而实现更准确的嘴部运动和结构的重建。

TalkingGaussian: 通过高斯点插值实现结构持续的三维说话头合成

TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via  Gaussian Splatting

Face animation has achieved much progress in computer vision. However,
prevailing GAN-based methods suffer from unnatural distortions and artifacts
due to sophisticated motion deformation. In this paper, we propose a Face
Animation framework with an attribute-guided Diffusion Model (FADM), which is
the first work to exploit the superior modeling capacity of diffusion models
for photo-realistic talking-head generation. To mitigate the uncontrollable
synthesis effect of the diffusion model, we design an Attribute-Guided
Conditioning Network (AGCN) to adaptively combine the coarse animation features
and 3D face reconstruction results, which can incorporate appearance and motion
conditions into the diffusion process. These specific designs help FADM rectify
unnatural artifacts and distortions, and also enrich high-fidelity facial
details through iterative diffusion refinements with accurate animation
attributes. FADM can flexibly and effectively improve existing animation
videos. Extensive experiments on widely used talking-head benchmarks validate
the effectiveness of FADM over prior arts.

该研究提出了一种基于属性引导扩散模型 (FADM) 的面部动画框架，该框架可逐步地改善非自然畸变和伪影。

基于属性引导扩散模型的面部动画

Face Animation with an Attribute-Guided Diffusion Model

We present a method that generates expressive talking heads from a single
facial image with audio as the only input. In contrast to previous approaches
that attempt to learn direct mappings from audio to raw pixels or points for
creating talking faces, our method first disentangles the content and speaker
information in the input audio signal. The audio content robustly controls the
motion of lips and nearby facial regions, while the speaker information
determines the specifics of facial expressions and the rest of the talking head
dynamics. Another key component of our method is the prediction of facial
landmarks reflecting speaker-aware dynamics. Based on this intermediate
representation, our method is able to synthesize photorealistic videos of
entire talking heads with full range of motion and also animate artistic
paintings, sketches, 2D cartoon characters, Japanese mangas, stylized
caricatures in a single unified framework. We present extensive quantitative
and qualitative evaluation of our method, in addition to user studies,
demonstrating generated talking heads of significantly higher quality compared
to prior state-of-the-art.

本文提出了一种从单个面部图像与仅含音频的输入生成富有表现力的谈话头像的方法，并能以单一的统一框架中合成艺术绘画、素描、2D 卡通角色、日本漫画和风格化漫画等图像，并通过定量和定性方法的评估以及用户研究，证明其比现有技术的生成头像的质量显着更高。