Speech-driven 3D facial animation has improved a lot recently while most
related works only utilize acoustic modality and neglect the influence of
visual and textual cues, leading to unsatisfactory results in terms of
precision and coherence. We argue that visual and textual cues are not trivial
information. Therefore, we present a novel framework, namely PMMTalk, using
complementary Pseudo Multi-Modal features for improving the accuracy of facial
animation. The framework entails three modules: PMMTalk encoder, cross-modal
alignment module, and PMMTalk decoder. Specifically, the PMMTalk encoder
employs the off-the-shelf talking head generation architecture and speech
recognition technology to extract visual and textual information from speech,
respectively. Subsequently, the cross-modal alignment module aligns the
audio-image-text features at temporal and semantic levels. Then PMMTalk decoder
is employed to predict lip-syncing facial blendshape coefficients. Contrary to
prior methods, PMMTalk only requires an additional random reference face image
but yields more accurate results. Additionally, it is artist-friendly as it
seamlessly integrates into standard animation production workflows by
introducing facial blendshape coefficients. Finally, given the scarcity of 3D
talking face datasets, we introduce a large-scale 3D Chinese Audio-Visual
Facial Animation (3D-CAVFA) dataset. Extensive experiments and user studies
show that our approach outperforms the state of the art. We recommend watching
the supplementary video.

PMMTalk 是一种使用伪多模态特征的新框架，通过跨模态对齐模块和 PMMTalk 解码器提高面部动画的准确性，同时引入面部混合形状系数，从而产生更准确的结果。

PMMTalk：基于互补伪多模特征的语音驱动 3D 面部动画

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo  Multi-modal Features

We present Blendshapes GHUM, an on-device ML pipeline that predicts 52 facial
blendshape coefficients at 30+ FPS on modern mobile phones, from a single
monocular RGB image and enables facial motion capture applications like virtual
avatars. Our main contributions are: i) an annotation-free offline method for
obtaining blendshape coefficients from real-world human scans, ii) a
lightweight real-time model that predicts blendshape coefficients based on
facial landmarks.

Blendshapes GHUM 是一个面向设备的机器学习流程，在现代移动手机上从单个单眼 RGB 图像中以 30+FPS 预测 52 个面部混合形状系数，并支持虚拟化头像等面部动作捕捉应用。我们的主要贡献是：i）一种无需注释的离线方法，可从现实世界的人体扫描获取混合形状系数，ii）基于面部标记预测混合形状系数的轻量级实时模型。