In the task of talking face generation, the objective is to generate a face
video with lips synchronized to the corresponding audio while preserving visual
details and identity information. Current methods face the challenge of
learning accurate lip synchronization while avoiding detrimental effects on
visual quality, as well as robustly evaluating such synchronization. To tackle
these problems, we propose utilizing an audio-visual speech representation
expert (AV-HuBERT) for calculating lip synchronization loss during training.
Moreover, leveraging AV-HuBERT's features, we introduce three novel lip
synchronization evaluation metrics, aiming to provide a comprehensive
assessment of lip synchronization performance. Experimental results, along with
a detailed ablation study, demonstrate the effectiveness of our approach and
the utility of the proposed evaluation metrics.

使用 AV-HuBERT 进行口型同步损失的计算，并引入三种新的口型同步评估指标，以提供全面的口型同步性能评估。实验结果和详细的消融研究表明了我们方法的有效性和提出的评估指标的实用性。