Speech-driven 3D facial animation is challenging due to the scarcity of
large-scale visual-audio datasets despite extensive research. Most prior works,
typically focused on learning regression models on a small dataset using the
method of least squares, encounter difficulties generating diverse lip
movements from speech and require substantial effort in refining the generated
outputs. To address these issues, we propose a speech-driven 3D facial
animation with a diffusion model (SAiD), a lightweight Transformer-based U-Net
with a cross-modality alignment bias between audio and visual to enhance lip
synchronization. Moreover, we introduce BlendVOCA, a benchmark dataset of pairs
of speech audio and parameters of a blendshape facial model, to address the
scarcity of public resources. Our experimental results demonstrate that the
proposed approach achieves comparable or superior performance in lip
synchronization to baselines, ensures more diverse lip movements, and
streamlines the animation editing process.

提出了一种使用扩散模型（SAiD）和基于 Transformer 的轻量级 U-Net 实现的语音驱动的 3D 面部动画方法，其中引入了音频和视觉之间交叉模态对齐偏差以增强唇部同步性，并通过引进 BlendVOCA 数据集解决公共资源稀缺问题。实验结果表明，该方法在唇部同步性上与基线方法相当或更出色，并确保了更多样化的唇部运动，并简化了动画编辑过程。