Speech-driven 3D facial animation is challenging due to the complex geometry
of human faces and the limited availability of 3D audio-visual data. Prior
works typically focus on learning phoneme-level features of short audio windows
with limited context, occasionally resulting in inaccurate lip movements. To
tackle this limitation, we propose a Transformer-based autoregressive model,
FaceFormer, which encodes the long-term audio context and autoregressively
predicts a sequence of animated 3D face meshes. To cope with the data scarcity
issue, we integrate the self-supervised pre-trained speech representations.
Also, we devise two biased attention mechanisms well suited to this specific
task, including the biased cross-modal multi-head (MH) attention and the biased
causal MH self-attention with a periodic positional encoding strategy. The
former effectively aligns the audio-motion modalities, whereas the latter
offers abilities to generalize to longer audio sequences. Extensive experiments
and a perceptual user study show that our approach outperforms the existing
state-of-the-arts. The code will be made available.

提出了一种基于 Transformer 的自回归模型，称为 FaceFormer，用于声控 3D 面部动画。该模型能够对长时间音频上下文进行编码，并自回归预测一系列动画 3D 面网格。通过自监督预训练语音表示，使得该模型能够成功解决数据稀缺问题。同时，设计了两种有偏的注意机制，并进行为这个特定任务进行优化。大量实验证明，该方法优于现有的最先进技术。