In this work, we re-think the task of speech enhancement in unconstrained
real-world environments. Current state-of-the-art methods use only the audio
stream and are limited in their performance in a wide range of real-world
noises. Recent works using lip movements as additional cues improve the quality
of generated speech over "audio-only" methods. But, these methods cannot be
used for several applications where the visual stream is unreliable or
completely absent. We propose a new paradigm for speech enhancement by
exploiting recent breakthroughs in speech-driven lip synthesis. Using one such
model as a teacher network, we train a robust student network to produce
accurate lip movements that mask away the noise, thus acting as a "visual noise
filter". The intelligibility of the speech enhanced by our pseudo-lip approach
is comparable (< 3% difference) to the case of using real lips. This implies
that we can exploit the advantages of using lip movements even in the absence
of a real video stream. We rigorously evaluate our model using quantitative
metrics as well as human evaluations. Additional ablation studies and a demo
video on our website containing qualitative comparisons and results clearly
illustrate the effectiveness of our approach. We provide a demo video which
clearly illustrates the effectiveness of our proposed approach on our website:
\url{this http URL}.
The code and models are also released for future research:
\url{this https URL}.

本文提出了一个基于说话者的唇部动作的噪音滤波器模型，该模型利用了最新的语音驱动唇部合成技术，能够在缺少视觉流的情况下优化语音质量和可懂性，并通过定量和人工评估证明了其有效性。