Radiance fields have demonstrated impressive performance in synthesizing
lifelike 3D talking heads. However, due to the difficulty in fitting steep
appearance changes, the prevailing paradigm that presents facial motions by
directly modifying point appearance may lead to distortions in dynamic regions.
To tackle this challenge, we introduce TalkingGaussian, a deformation-based
radiance fields framework for high-fidelity talking head synthesis. Leveraging
the point-based Gaussian Splatting, facial motions can be represented in our
method by applying smooth and continuous deformations to persistent Gaussian
primitives, without requiring to learn the difficult appearance change like
previous methods. Due to this simplification, precise facial motions can be
synthesized while keeping a highly intact facial feature. Under such a
deformation paradigm, we further identify a face-mouth motion inconsistency
that would affect the learning of detailed speaking motions. To address this
conflict, we decompose the model into two branches separately for the face and
inside mouth areas, therefore simplifying the learning tasks to help
reconstruct more accurate motion and structure of the mouth region. Extensive
experiments demonstrate that our method renders high-quality lip-synchronized
talking head videos, with better facial fidelity and higher efficiency compared
with previous methods.

本研究提出了一种基于变形的辐射场框架 TalkingGaussian，用于高保真度语音同步头像合成，并通过将模型分为两个分支来解决面部和口内区域的运动不一致问题，从而实现更准确的嘴部运动和结构的重建。