Speech-driven 3D facial animation technology has been developed for years,
but its practical application still lacks expectations. The main challenges lie
in data limitations, lip alignment, and the naturalness of facial expressions.
Although lip alignment has seen many related studies, existing methods struggle
to synthesize natural and realistic expressions, resulting in a mechanical and
stiff appearance of facial animations. Even with some research extracting
emotional features from speech, the randomness of facial movements limits the
effective expression of emotions. To address this issue, this paper proposes a
method called CSTalk (Correlation Supervised) that models the correlations
among different regions of facial movements and supervises the training of the
generative model to generate realistic expressions that conform to human facial
motion patterns. To generate more intricate animations, we employ a rich set of
control parameters based on the metahuman character model and capture a dataset
for five different emotions. We train a generative network using an autoencoder
structure and input an emotion embedding vector to achieve the generation of
user-control expressions. Experimental results demonstrate that our method
outperforms existing state-of-the-art methods.

本研究提出了一种名为 CSTalk 的方法，通过建模面部运动不同区域之间的相关性并监督生成模型的训练，生成符合人脸运动模式的真实表情，解决了语音驱动的 3D 面部动画技术中面临的数据限制、唇部对齐以及面部表情自然度等挑战，实验结果表明我们的方法优于现有最先进的方法。