In this work, we investigate the problem of lip-syncing a talking face video
of an arbitrary identity to match a target speech segment. Current works excel
at producing accurate lip movements on a static image or videos of specific
people seen during the training phase. However, they fail to accurately morph
the lip movements of arbitrary identities in dynamic, unconstrained talking
face videos, resulting in significant parts of the video being out-of-sync with
the new audio. We identify key reasons pertaining to this and hence resolve
them by learning from a powerful lip-sync discriminator. Next, we propose new,
rigorous evaluation benchmarks and metrics to accurately measure lip
synchronization in unconstrained videos. Extensive quantitative evaluations on
our challenging benchmarks show that the lip-sync accuracy of the videos
generated by our Wav2Lip model is almost as good as real synced videos. We
provide a demo video clearly showing the substantial impact of our Wav2Lip
model and evaluation benchmarks on our website:
\url{cvit.iiit.ac.in/research/projects/cvit-projects/a-lip-sync-expert-is-all-you-need-for-speech-to-lip-generation-in-the-wild}.
The code and models are released at this GitHub repository:
\url{github.com/Rudrabha/Wav2Lip}. You can also try out the interactive demo at
this link: \url{bhaasha.iiit.ac.in/lipsync}.

研究如何 lip-sync 一个任意身份的说话面部视频以匹配目标语音片段，并使用 Wav2Lip 模型和新的评估基准，成功地在自由视频中实现了较好的 lip-sync 精度。

野外语音到唇形生成只需要一个口型同步专家

A Lip Sync Expert Is All You Need for Speech to Lip Generation In The  Wild

Although deep learning approaches have achieved performance surpassing humans
for still image-based face recognition, unconstrained video-based face
recognition is still a challenging task due to large volume of data to be
processed and intra/inter-video variations on pose, illumination, occlusion,
scene, blur, video quality, etc. In this work, we consider challenging
scenarios for unconstrained video-based face recognition from multiple-shot
videos and surveillance videos with low-quality frames. To handle these
problems, we propose a robust and efficient system for unconstrained
video-based face recognition, which is composed of modules for face/fiducial
detection, face association, and face recognition. First, we use multi-scale
single-shot face detectors to efficiently localize faces in videos. The
detected faces are then grouped respectively through carefully designed face
association methods, especially for multi-shot videos. Finally, the faces are
recognized by the proposed face matcher based on an unsupervised subspace
learning approach and a subspace-to-subspace similarity metric. Extensive
experiments on challenging video datasets, such as Multiple Biometric Grand
Challenge (MBGC), Face and Ocular Challenge Series (FOCS), IARPA Janus
Surveillance Video Benchmark (IJB-S) for low-quality surveillance videos and
IARPA JANUS Benchmark B (IJB-B) for multiple-shot videos, demonstrate that the
proposed system can accurately detect and associate faces from unconstrained
videos and effectively learn robust and discriminative features for
recognition.

提出了一个用于多个挑战性情景的视频人脸识别的鲁棒且高效的系统，包括人脸 / 标志检测、人脸关联和人脸识别等模块，特别是针对多镜头视频设计了精心的人脸关联方法，采用一种基于无监督子空间学习方法和子空间到子空间相似度度量的人脸匹配器来识别人脸，广泛实验表明该系统可以准确地检测和关联未约束的视频中的人脸，并有效地学习到具有鲁棒性和区分性的特征用于识别。