The generation of stylistic 3D facial animations driven by speech poses a
significant challenge as it requires learning a many-to-many mapping between
speech, style, and the corresponding natural facial motion. However, existing
methods either employ a deterministic model for speech-to-motion mapping or
encode the style using a one-hot encoding scheme. Notably, the one-hot encoding
approach fails to capture the complexity of the style and thus limits
generalization ability. In this paper, we propose DiffPoseTalk, a generative
framework based on the diffusion model combined with a style encoder that
extracts style embeddings from short reference videos. During inference, we
employ classifier-free guidance to guide the generation process based on the
speech and style. We extend this to include the generation of head poses,
thereby enhancing user perception. Additionally, we address the shortage of
scanned 3D talking face data by training our model on reconstructed 3DMM
parameters from a high-quality, in-the-wild audio-visual dataset. Our extensive
experiments and user study demonstrate that our approach outperforms
state-of-the-art methods. The code and dataset will be made publicly available.

这篇论文提出了 DiffPoseTalk，一种基于扩散模型和风格编码器的生成框架，该框架通过从短参考视频中提取风格嵌入来辅助面部动画生成，并通过利用语音和风格进行生成过程的指导，进而提高用户感知。此外，作者还通过对高质量、真实场景下的音频 - 视觉数据集中重建的 3DMM 参数进行训练，解决了扫描 3D 说话脸数据不足的问题。丰富的实验和用户研究表明，该方法优于现有的方法。将会公开发布代码和数据集。