Speech-driven facial animation methods usually contain two main classes, 3D
and 2D talking face, both of which attract considerable research attention in
recent years. However, to the best of our knowledge, the research on 3D talking
face does not go deeper as 2D talking face, in the aspect of
lip-synchronization (lip-sync) and speech perception. To mind the gap between
the two sub-fields, we propose a learning framework named Learn2Talk, which can
construct a better 3D talking face network by exploiting two expertise points
from the field of 2D talking face. Firstly, inspired by the audio-video sync
network, a 3D sync-lip expert model is devised for the pursuit of lip-sync
between audio and 3D facial motion. Secondly, a teacher model selected from 2D
talking face methods is used to guide the training of the audio-to-3D motions
regression network to yield more 3D vertex accuracy. Extensive experiments show
the advantages of the proposed framework in terms of lip-sync, vertex accuracy
and speech perception, compared with state-of-the-arts. Finally, we show two
applications of the proposed framework: audio-visual speech recognition and
speech-driven 3D Gaussian Splatting based avatar animation.

提出了一种名为 Learn2Talk 的学习框架，该框架可以通过利用 2D 说话面领域的两个专业点来构建更好的 3D 说话面网络，主要关注嘴唇同步、语音感知、音频 - 3D 动作回归网络等方面的研究。

Learn2Talk: 3D 口型学习 2D 口型

Learn2Talk: 3D Talking Face Learns from 2D Talking Face

Speech-driven 3D facial animation has been widely studied, yet there is still
a gap to achieving realism and vividness due to the highly ill-posed nature and
scarcity of audio-visual data. Existing works typically formulate the
cross-modal mapping into a regression task, which suffers from the
regression-to-mean problem leading to over-smoothed facial motions. In this
paper, we propose to cast speech-driven facial animation as a code query task
in a finite proxy space of the learned codebook, which effectively promotes the
vividness of the generated motions by reducing the cross-modal mapping
uncertainty. The codebook is learned by self-reconstruction over real facial
motions and thus embedded with realistic facial motion priors. Over the
discrete motion space, a temporal autoregressive model is employed to
sequentially synthesize facial motions from the input speech signal, which
guarantees lip-sync as well as plausible facial expressions. We demonstrate
that our approach outperforms current state-of-the-art methods both
qualitatively and quantitatively. Also, a user study further justifies our
superiority in perceptual quality.

本文提出将基于语音的面部动画转化为在学习到的码本的有限代理空间中的代码查询任务，有效地减少了跨模态映射的不确定性，从而提高了生成的运动的生动性，并且通过使用一个自动回归模型，演示了我们方法在视觉和感知质量上均优于当前最先进的方法。

CodeTalker: 利用离散运动先验驱动的语音驱动 3D 面部动画

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Speech-driven facial animation is the process that automatically synthesizes
talking characters based on speech signals. The majority of work in this domain
creates a mapping from audio features to visual features. This approach often
requires post-processing using computer graphics techniques to produce
realistic albeit subject dependent results. We present an end-to-end system
that generates videos of a talking head, using only a still image of a person
and an audio clip containing speech, without relying on handcrafted
intermediate features. Our method generates videos which have (a) lip movements
that are in sync with the audio and (b) natural facial expressions such as
blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on
achieving detailed frames, audio-visual synchronization, and realistic
expressions. We quantify the contribution of each component in our model using
an ablation study and we provide insights into the latent representation of the
model. The generated videos are evaluated based on sharpness, reconstruction
quality, lip-reading accuracy, synchronization as well as their ability to
generate natural blinks.

本文介绍了一种基于语音信号自动生成会说话的角色的方法，该方法使用了端到端的系统并使用 Generative Adversarial Network (生成对抗网络) 来确保视频与音频的实时同步，并且产生了自然的面部表情。

利用 GAN 实现逼真的语音驱动面部动画

Realistic Speech-Driven Facial Animation with GANs

Speech-driven facial animation is the process which uses speech signals to
automatically synthesize a talking character. The majority of work in this
domain creates a mapping from audio features to visual features. This often
requires post-processing using computer graphics techniques to produce
realistic albeit subject dependent results. We present a system for generating
videos of a talking head, using a still image of a person and an audio clip
containing speech, that doesn't rely on any handcrafted intermediate features.
To the best of our knowledge, this is the first method capable of generating
subject independent realistic videos directly from raw audio. Our method can
generate videos which have (a) lip movements that are in sync with the audio
and (b) natural facial expressions such as blinks and eyebrow movements. We
achieve this by using a temporal GAN with 2 discriminators, which are capable
of capturing different aspects of the video. The effect of each component in
our system is quantified through an ablation study. The generated videos are
evaluated based on their sharpness, reconstruction quality, and lip-reading
accuracy. Finally, a user study is conducted, confirming that temporal GANs
lead to more natural sequences than a static GAN-based approach.

使用时间依赖的 GAN 模型和音视频同步的方法，实现了从原始音频和人物图像生成高质量且与个体无关的自然面部表情和嘴唇运动的视频。