In this work, we address the problem of generating speech from silent lip
videos for any speaker in the wild. In stark contrast to previous works, our
method (i) is not restricted to a fixed number of speakers, (ii) does not
explicitly impose constraints on the domain or the vocabulary and (iii) deals
with videos that are recorded in the wild as opposed to within laboratory
settings. The task presents a host of challenges, with the key one being that
many features of the desired target speech, like voice, pitch and linguistic
content, cannot be entirely inferred from the silent face video. In order to
handle these stochastic variations, we propose a new VAE-GAN architecture that
learns to associate the lip and speech sequences amidst the variations. With
the help of multiple powerful discriminators that guide the training process,
our generator learns to synthesize speech sequences in any voice for the lip
movements of any person. Extensive experiments on multiple datasets show that
we outperform all baselines by a large margin. Further, our network can be
fine-tuned on videos of specific identities to achieve a performance comparable
to single-speaker models that are trained on $4\times$ more data. We conduct
numerous ablation studies to analyze the effect of different modules of our
architecture. We also provide a demo video that demonstrates several
qualitative results along with the code and trained models on our website:
http://cvit.iiit.ac.in/research/projects/cvit-projects/lip-to-speech-synthesis}

采用新型的 VAE-GAN 架构生成多说话者在野外拍摄的沉默唇视频中的语音，比其他基线模型表现更好，能够实现针对特定身份的微调和生成不同语音的语音序列。

野外任意说话人唇语合成

Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

We present the Tongue and Lips corpus (TaL), a multi-speaker corpus of audio,
ultrasound tongue imaging, and lip videos. TaL consists of two parts: TaL1 is a
set of six recording sessions of one professional voice talent, a male native
speaker of English; TaL80 is a set of recording sessions of 81 native speakers
of English without voice talent experience. Overall, the corpus contains 24
hours of parallel ultrasound, video, and audio data, of which approximately
13.5 hours are speech. This paper describes the corpus and presents benchmark
results for the tasks of speech recognition, speech synthesis
(articulatory-to-acoustic mapping), and automatic synchronisation of ultrasound
to audio. The TaL corpus is publicly available under the CC BY-NC 4.0 license.

本文介绍了 Tongue and Lips 语料库（TaL）的数据来源和处理方式，并在多个任务上对该语料库进行了基准测试，结果表明 TaL 可以用于语音识别和语音合成等任务。