Talking face generation, also known as speech-to-lip generation, reconstructs
facial motions concerning lips given coherent speech input. The previous
studies revealed the importance of lip-speech synchronization and visual
quality. Despite much progress, they hardly focus on the content of lip
movements i.e., the visual intelligibility of the spoken words, which is an
important aspect of generation quality. To address the problem, we propose
using a lip-reading expert to improve the intelligibility of the generated lip
regions by penalizing the incorrect generation results. Moreover, to compensate
for data scarcity, we train the lip-reading expert in an audio-visual
self-supervised manner. With a lip-reading expert, we propose a novel
contrastive learning to enhance lip-speech synchronization, and a transformer
to encode audio synchronically with video, while considering global temporal
dependency of audio. For evaluation, we propose a new strategy with two
different lip-reading experts to measure intelligibility of the generated
videos. Rigorous experiments show that our proposal is superior to other
State-of-the-art (SOTA) methods, such as Wav2Lip, in reading intelligibility
i.e., over 38% Word Error Rate (WER) on LRS2 dataset and 27.8% accuracy on LRW
dataset. We also achieve the SOTA performance in lip-speech synchronization and
comparable performances in visual quality.

本文提出了一种利用口语专家、对不正确的生成结果进行惩罚和全局时间和视觉同步编码的对比学习和变压器方法来提高口语智能理解度的方法，并使用两种不同的口语专家评估生成视频的智能理解度。我们的方法在读取可理解度、嘴唇运动同步等方面优于当前的一些最先进方法，并获得其他方面的收益。