Speech-driven 3D facial animation is important for many multimedia
applications. Recent work has shown promise in using either Diffusion models or
Transformer architectures for this task. However, their mere aggregation does
not lead to improved performance. We suspect this is due to a shortage of
paired audio-4D data, which is crucial for the Transformer to effectively
perform as a denoiser within the Diffusion framework. To tackle this issue, we
present DiffSpeaker, a Transformer-based network equipped with novel biased
conditional attention modules. These modules serve as substitutes for the
traditional self/cross-attention in standard Transformers, incorporating
thoughtfully designed biases that steer the attention mechanisms to concentrate
on both the relevant task-specific and diffusion-related conditions. We also
explore the trade-off between accurate lip synchronization and non-verbal
facial expressions within the Diffusion paradigm. Experiments show our model
not only achieves state-of-the-art performance on existing benchmarks, but also
fast inference speed owing to its ability to generate facial motions in
parallel.

使用 DiffSpeaker 网络，该网络具备新颖的有偏条件注意力模块，用于聚焦任务相关和扩散相关的条件，从而解决传统的 Diffusion 模型和 Transformer 架构在语音驱动的 3D 面部动画生成中性能提升有限的问题。我们的模型在现有基准测试中不仅达到了最先进的性能，而且由于能够并行生成面部动作，具备快速推理的速度。