Person-generic audio-driven face generation is a challenging task in computer
vision. Previous methods have achieved remarkable progress in audio-visual
synchronization, but there is still a significant gap between current results
and practical applications. The challenges are two-fold: 1) Preserving unique
individual traits for achieving high-precision lip synchronization. 2)
Generating high-quality facial renderings in real-time performance. In this
paper, we propose a novel generalized audio-driven framework RealTalk, which
consists of an audio-to-expression transformer and a high-fidelity
expression-to-face renderer. In the first component, we consider both identity
and intra-personal variation features related to speaking lip movements. By
incorporating cross-modal attention on the enriched facial priors, we can
effectively align lip movements with audio, thus attaining greater precision in
expression prediction. In the second component, we design a lightweight facial
identity alignment (FIA) module which includes a lip-shape control structure
and a face texture reference structure. This novel design allows us to generate
fine details in real-time, without depending on sophisticated and inefficient
feature alignment modules. Our experimental results, both quantitative and
qualitative, on public datasets demonstrate the clear advantages of our method
in terms of lip-speech synchronization and generation quality. Furthermore, our
method is efficient and requires fewer computational resources, making it
well-suited to meet the needs of practical applications.

该论文提出了一个新颖的通用音频驱动框架 RealTalk，包括音频到表情转换和表情到人脸渲染两个组成部分，通过跨模态注意力对丰富的面部先验信息进行对齐，以实现高精确度的唇语同步和实时生成高质量的面部图像。该方法在公共数据集上进行的实验结果表明，它在唇语同步和生成质量方面具有明显的优势，并且具有高效且需要较少计算资源的特点，非常适合实际应用。