In this paper, we introduce a novel approach to address the task of
synthesizing speech from silent videos of any in-the-wild speaker solely based
on lip movements. The traditional approach of directly generating speech from
lip videos faces the challenge of not being able to learn a robust language
model from speech alone, resulting in unsatisfactory outcomes. To overcome this
issue, we propose incorporating noisy text supervision using a state-of-the-art
lip-to-text network that instills language information into our model. The
noisy text is generated using a pre-trained lip-to-text model, enabling our
approach to work without text annotations during inference. We design a visual
text-to-speech network that utilizes the visual stream to generate accurate
speech, which is in-sync with the silent input video. We perform extensive
experiments and ablation studies, demonstrating our approach's superiority over
the current state-of-the-art methods on various benchmark datasets. Further, we
demonstrate an essential practical application of our method in assistive
technology by generating speech for an ALS patient who has lost the voice but
can make mouth movements. Our demo video, code, and additional details can be
found at
http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw.

提出了一种从任何野外说话人的无声视频中仅基于嘴唇动作合成语音的新方法，通过将嘴唇到文本网络的嘈杂文本监督纳入模型中实现了语言信息注入，并使用视觉流生成与输入视频同步的准确语音，通过广泛的实验和消融研究表明了该方法在各种基准数据集上的优越性，并在辅助技术中展示了其重要的实际应用。