We propose GaussianTalker, a novel framework for real-time generation of
pose-controllable talking heads. It leverages the fast rendering capabilities
of 3D Gaussian Splatting (3DGS) while addressing the challenges of directly
controlling 3DGS with speech audio. GaussianTalker constructs a canonical 3DGS
representation of the head and deforms it in sync with the audio. A key insight
is to encode the 3D Gaussian attributes into a shared implicit feature
representation, where it is merged with audio features to manipulate each
Gaussian attribute. This design exploits the spatial-aware features and
enforces interactions between neighboring points. The feature embeddings are
then fed to a spatial-audio attention module, which predicts frame-wise offsets
for the attributes of each Gaussian. It is more stable than previous
concatenation or multiplication approaches for manipulating the numerous
Gaussians and their intricate parameters. Experimental results showcase
GaussianTalker's superiority in facial fidelity, lip synchronization accuracy,
and rendering speed compared to previous methods. Specifically, GaussianTalker
achieves a remarkable rendering speed of 120 FPS, surpassing previous
benchmarks. Our code is made available at
this https URL .

提出了 GaussianTalker，这是一个实时生成可调控姿势的会话头部的新框架，它利用 3D 高斯喷射（3DGS）的快速渲染能力，同时解决了直接控制 3DGS 与语音音频的挑战。