In this work, we address the problem of generating speech from silent lip
videos for any speaker in the wild. In stark contrast to previous works, our
method (i) is not restricted to a fixed number of speakers, (ii) does not
explicitly impose constraints on the domain or the vocabulary and (iii) deals
with videos that are recorded in the wild as opposed to within laboratory
settings. The task presents a host of challenges, with the key one being that
many features of the desired target speech, like voice, pitch and linguistic
content, cannot be entirely inferred from the silent face video. In order to
handle these stochastic variations, we propose a new VAE-GAN architecture that
learns to associate the lip and speech sequences amidst the variations. With
the help of multiple powerful discriminators that guide the training process,
our generator learns to synthesize speech sequences in any voice for the lip
movements of any person. Extensive experiments on multiple datasets show that
we outperform all baselines by a large margin. Further, our network can be
fine-tuned on videos of specific identities to achieve a performance comparable
to single-speaker models that are trained on $4\times$ more data. We conduct
numerous ablation studies to analyze the effect of different modules of our
architecture. We also provide a demo video that demonstrates several
qualitative results along with the code and trained models on our website:
http://cvit.iiit.ac.in/research/projects/cvit-projects/lip-to-speech-synthesis}

采用新型的 VAE-GAN 架构生成多说话者在野外拍摄的沉默唇视频中的语音，比其他基线模型表现更好，能够实现针对特定身份的微调和生成不同语音的语音序列。