Speech-driven facial animation methods usually contain two main classes, 3D
and 2D talking face, both of which attract considerable research attention in
recent years. However, to the best of our knowledge, the research on 3D talking
face does not go deeper as 2D talking face, in the aspect of
lip-synchronization (lip-sync) and speech perception. To mind the gap between
the two sub-fields, we propose a learning framework named Learn2Talk, which can
construct a better 3D talking face network by exploiting two expertise points
from the field of 2D talking face. Firstly, inspired by the audio-video sync
network, a 3D sync-lip expert model is devised for the pursuit of lip-sync
between audio and 3D facial motion. Secondly, a teacher model selected from 2D
talking face methods is used to guide the training of the audio-to-3D motions
regression network to yield more 3D vertex accuracy. Extensive experiments show
the advantages of the proposed framework in terms of lip-sync, vertex accuracy
and speech perception, compared with state-of-the-arts. Finally, we show two
applications of the proposed framework: audio-visual speech recognition and
speech-driven 3D Gaussian Splatting based avatar animation.

提出了一种名为 Learn2Talk 的学习框架，该框架可以通过利用 2D 说话面领域的两个专业点来构建更好的 3D 说话面网络，主要关注嘴唇同步、语音感知、音频 - 3D 动作回归网络等方面的研究。