Audio-driven one-shot talking face generation methods are usually trained on
video resources of various persons. However, their created videos often suffer
unnatural mouth shapes and asynchronous lips because those methods struggle to
learn a consistent speech style from different speakers. We observe that it
would be much easier to learn a consistent speech style from a specific
speaker, which leads to authentic mouth movements. Hence, we propose a novel
one-shot talking face generation framework by exploring consistent correlations
between audio and visual motions from a specific speaker and then transferring
audio-driven motion fields to a reference image. Specifically, we develop an
Audio-Visual Correlation Transformer (AVCT) that aims to infer talking motions
represented by keypoint based dense motion fields from an input audio. In
particular, considering audio may come from different identities in deployment,
we incorporate phonemes to represent audio signals. In this manner, our AVCT
can inherently generalize to audio spoken by other identities. Moreover, as
face keypoints are used to represent speakers, AVCT is agnostic against
appearances of the training speaker, and thus allows us to manipulate face
images of different identities readily. Considering different face shapes lead
to different motions, a motion field transfer module is exploited to reduce the
audio-driven dense motion field gap between the training identity and the
one-shot reference. Once we obtained the dense motion field of the reference
image, we employ an image renderer to generate its talking face videos from an
audio clip. Thanks to our learned consistent speaking style, our method
generates authentic mouth shapes and vivid movements. Extensive experiments
demonstrate that our synthesized videos outperform the state-of-the-art in
terms of visual quality and lip-sync.

本文提出了一种基于 AVCT 的新型一次性说话脸部生成框架，通过从特定说话者的音频和视觉动作中探索一致的相关性，然后将基于密集运动场的音频驱动的运动场转移至参考图像，以生成具有真实嘴部形状和生动运动的视频。