Recent methods for audio-driven talking head synthesis often optimize neural
radiance fields (NeRF) on a monocular talking portrait video, leveraging its
capability to render high-fidelity and 3D-consistent novel-view frames.
However, they often struggle to reconstruct complete face geometry due to the
absence of comprehensive 3D information in the input monocular videos. In this
paper, we introduce a novel audio-driven talking head synthesis framework,
called Talk3D, that can faithfully reconstruct its plausible facial geometries
by effectively adopting the pre-trained 3D-aware generative prior. Given the
personalized 3D generative model, we present a novel audio-guided attention
U-Net architecture that predicts the dynamic face variations in the NeRF space
driven by audio. Furthermore, our model is further modulated by audio-unrelated
conditioning tokens which effectively disentangle variations unrelated to audio
features. Compared to existing methods, our method excels in generating
realistic facial geometries even under extreme head poses. We also conduct
extensive experiments showing our approach surpasses state-of-the-art
benchmarks in terms of both quantitative and qualitative evaluations.

Talk3D 是一种新的音频驱动的说话头合成框架，通过有效地采用预训练的 3D 感知生成先验模型，可以忠实地重建其合理的面部几何形状。通过音频驱动的注意力 U-Net 架构，我们的模型预测 NeRF 空间中由音频驱动的动态面部变化，并由与音频无关的调节令牌调制，有效地解耦与音频特征无关的变化。与现有方法相比，我们的方法在生成逼真的面部几何形状方面表现出色，即使在极端的头部姿势下也能如此。我们还进行了广泛的实验证明我们的方法在定量和定性评估方面超越了现有的最新基准。

Talk3D：个性化 3D 生成先验的高保真说话肖像合成

Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D  Generative Prior

Artificial data synthesis is currently a well studied topic with useful
applications in data science, computer vision, graphics and many other fields.
Generating realistic data is especially challenging since human perception is
highly sensitive to non realistic appearance. In recent times, new levels of
realism have been achieved by advances in GAN training procedures and
architectures. These successful models, however, are tuned mostly for use with
regularly sampled data such as images, audio and video. Despite the successful
application of the architecture on these types of media, applying the same
tools to geometric data poses a far greater challenge. The study of geometric
deep learning is still a debated issue within the academic community as the
lack of intrinsic parametrization inherent to geometric objects prohibits the
direct use of convolutional filters, a main building block of today's machine
learning systems. In this paper we propose a new method for generating
realistic human facial geometries coupled with overlayed textures. We
circumvent the parametrization issue by imposing a global mapping from our data
to the unit rectangle. We further discuss how to design such a mapping to
control the mapping distortion and conserve area within the mapped image. By
representing geometric textures and geometries as images, we are able to use
advanced GAN methodologies to generate new geometries. We address the often
neglected topic of relation between texture and geometry and propose to use
this correlation to match between generated textures and their corresponding
geometries. We offer a new method for training GAN models on partially
corrupted data. Finally, we provide empirical evidence demonstrating our
generative model's ability to produce examples of new identities independent
from the training data while maintaining a high level of realism, two traits
that are often at odds.

提出新的生成现实人脸几何结构和纹理重叠的方法，通过将几何纹理表示为图像并将其映射到单位矩形上，绕过了几何数据固有的参数化问题，使用先进的 GAN 方法生成新的几何结构，并提出将纹理和几何之间的关系匹配来保持高质量的真实感，证明了该方法的生成模型具有独立于训练数据的新身份的能力。