The synthesis of 3D facial animations from speech has garnered considerable
attention. Due to the scarcity of high-quality 4D facial data and
well-annotated abundant multi-modality labels, previous methods often suffer
from limited realism and a lack of lexible conditioning. We address this
challenge through a trilogy. We first introduce Generalized Neural Parametric
Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
geometry and images to a highly generalized expression latent space, decoupling
expressions and identities. Then, we utilize GNPFA to extract high-quality
expressions and accurate head poses from a large array of videos. This presents
the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
animation dataset with well-annotated emotional and style labels. Finally, we
propose Media2Face, a diffusion model in GNPFA latent space for co-speech
facial animation generation, accepting rich multi-modality guidances from
audio, text, and image. Extensive experiments demonstrate that our model not
only achieves high fidelity in facial animation synthesis but also broadens the
scope of expressiveness and style adaptability in 3D facial animation.

从语音中合成 3D 面部动画引起了极大的关注。我们通过三位一体的方法来应对这一挑战，首先介绍了广义神经参数面部特征 (GNPFA)，然后利用 GNPFA 从大量的视频中提取高质量的表情和准确的头部姿势，最后，我们提出了 Media2Face，这是一个在 GNPFA 潜在空间中的扩散模型，用于生成与音频、文本和图像相关的共语面部动画。大量实验表明，我们的模型不仅在面部动画合成方面具有高保真性，还扩大了 3D 面部动画的表现力和风格适应性。