Talking face generation aims to synthesize a face video with precise lip
synchronization as well as a smooth transition of facial motion over the entire
video via the given speech clip and facial image. Most existing methods mainly
focus on either disentangling the information in a single image or learning
temporal information between frames. However, cross-modality coherence between
audio and video information has not been well addressed during synthesis. In
this paper, we propose a novel arbitrary talking face generation framework by
discovering the audio-visual coherence via the proposed Asymmetric Mutual
Information Estimator (AMIE). In addition, we propose a Dynamic Attention (DA)
block by selectively focusing the lip area of the input image during the
training stage, to further enhance lip synchronization. Experimental results on
benchmark LRW dataset and GRID dataset transcend the state-of-the-art methods
on prevalent metrics with robust high-resolution synthesizing on gender and
pose variations.

本文提出了一种新的任意说话脸生成框架，通过提出的 AMIE 发现音频和视频信息之间的视听一致性，以及通过训练阶段中选择性聚焦输入图像的嘴唇区域来进一步增强唇部同步。在 LRW 数据集和 GRID 数据集上进行的实验结果显示，该方法在性别和姿势变化方面具有鲁棒的高分辨率综合，改进了现有方法在普遍指标上的性能。