Speech-driven 3D face animation technique, extending its applications to
various multimedia fields. Previous research has generated promising realistic
lip movements and facial expressions from audio signals. However, traditional
regression models solely driven by data face several essential problems, such
as difficulties in accessing precise labels and domain gaps between different
modalities, leading to unsatisfactory results lacking precision and coherence.
To enhance the visual accuracy of generated lip movement while reducing the
dependence on labeled data, we propose a novel framework SelfTalk, by involving
self-supervision in a cross-modals network system to learn 3D talking faces.
The framework constructs a network system consisting of three modules: facial
animator, speech recognizer, and lip-reading interpreter. The core of SelfTalk
is a commutative training diagram that facilitates compatible features exchange
among audio, text, and lip shape, enabling our models to learn the intricate
connection between these factors. The proposed framework leverages the
knowledge learned from the lip-reading interpreter to generate more plausible
lip shapes. Extensive experiments and user studies demonstrate that our
proposed approach achieves state-of-the-art performance both qualitatively and
quantitatively. We recommend watching the supplementary video.

通过引入自我监督的跨模态网络系统，学习了三维说话脸部的 SelfTalk 框架构建了一个由三个模块组成的网络系统，以提高口型生成的精度和准确性。