Recent methods for audio-driven talking head synthesis often optimize neural
radiance fields (NeRF) on a monocular talking portrait video, leveraging its
capability to render high-fidelity and 3D-consistent novel-view frames.
However, they often struggle to reconstruct complete face geometry due to the
absence of comprehensive 3D information in the input monocular videos. In this
paper, we introduce a novel audio-driven talking head synthesis framework,
called Talk3D, that can faithfully reconstruct its plausible facial geometries
by effectively adopting the pre-trained 3D-aware generative prior. Given the
personalized 3D generative model, we present a novel audio-guided attention
U-Net architecture that predicts the dynamic face variations in the NeRF space
driven by audio. Furthermore, our model is further modulated by audio-unrelated
conditioning tokens which effectively disentangle variations unrelated to audio
features. Compared to existing methods, our method excels in generating
realistic facial geometries even under extreme head poses. We also conduct
extensive experiments showing our approach surpasses state-of-the-art
benchmarks in terms of both quantitative and qualitative evaluations.

Talk3D 是一种新的音频驱动的说话头合成框架，通过有效地采用预训练的 3D 感知生成先验模型，可以忠实地重建其合理的面部几何形状。通过音频驱动的注意力 U-Net 架构，我们的模型预测 NeRF 空间中由音频驱动的动态面部变化，并由与音频无关的调节令牌调制，有效地解耦与音频特征无关的变化。与现有方法相比，我们的方法在生成逼真的面部几何形状方面表现出色，即使在极端的头部姿势下也能如此。我们还进行了广泛的实验证明我们的方法在定量和定性评估方面超越了现有的最新基准。