The synthesis of 3D facial animations from speech has garnered considerable
attention. Due to the scarcity of high-quality 4D facial data and
well-annotated abundant multi-modality labels, previous methods often suffer
from limited realism and a lack of lexible conditioning. We address this
challenge through a trilogy. We first introduce Generalized Neural Parametric
Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
geometry and images to a highly generalized expression latent space, decoupling
expressions and identities. Then, we utilize GNPFA to extract high-quality
expressions and accurate head poses from a large array of videos. This presents
the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
animation dataset with well-annotated emotional and style labels. Finally, we
propose Media2Face, a diffusion model in GNPFA latent space for co-speech
facial animation generation, accepting rich multi-modality guidances from
audio, text, and image. Extensive experiments demonstrate that our model not
only achieves high fidelity in facial animation synthesis but also broadens the
scope of expressiveness and style adaptability in 3D facial animation.

从语音中合成 3D 面部动画引起了极大的关注。我们通过三位一体的方法来应对这一挑战，首先介绍了广义神经参数面部特征 (GNPFA)，然后利用 GNPFA 从大量的视频中提取高质量的表情和准确的头部姿势，最后，我们提出了 Media2Face，这是一个在 GNPFA 潜在空间中的扩散模型，用于生成与音频、文本和图像相关的共语面部动画。大量实验表明，我们的模型不仅在面部动画合成方面具有高保真性，还扩大了 3D 面部动画的表现力和风格适应性。

Media2Face: 借助多模态引导生成共话脸部动画

Media2Face: Co-speech Facial Animation Generation With Multi-Modality  Guidance

The generation of stylistic 3D facial animations driven by speech poses a
significant challenge as it requires learning a many-to-many mapping between
speech, style, and the corresponding natural facial motion. However, existing
methods either employ a deterministic model for speech-to-motion mapping or
encode the style using a one-hot encoding scheme. Notably, the one-hot encoding
approach fails to capture the complexity of the style and thus limits
generalization ability. In this paper, we propose DiffPoseTalk, a generative
framework based on the diffusion model combined with a style encoder that
extracts style embeddings from short reference videos. During inference, we
employ classifier-free guidance to guide the generation process based on the
speech and style. We extend this to include the generation of head poses,
thereby enhancing user perception. Additionally, we address the shortage of
scanned 3D talking face data by training our model on reconstructed 3DMM
parameters from a high-quality, in-the-wild audio-visual dataset. Our extensive
experiments and user study demonstrate that our approach outperforms
state-of-the-art methods. The code and dataset will be made publicly available.

这篇论文提出了 DiffPoseTalk，一种基于扩散模型和风格编码器的生成框架，该框架通过从短参考视频中提取风格嵌入来辅助面部动画生成，并通过利用语音和风格进行生成过程的指导，进而提高用户感知。此外，作者还通过对高质量、真实场景下的音频 - 视觉数据集中重建的 3DMM 参数进行训练，解决了扫描 3D 说话脸数据不足的问题。丰富的实验和用户研究表明，该方法优于现有的方法。将会公开发布代码和数据集。