Visual speech enhancement is used on videos shot in noisy environments to enhance the voice of a visible speaker and to reduce background noise. While most existing methods use audio-only inputs, we propose an audio-visual neural network model for this purpose. The visible mouth movements are used to separate the speaker's voice from the background sounds. Instead of training our speech enhancement model on a wide range of possible noise types, we train the model on videos where other speech samples of the target speaker are used as background noise. A model trained using this paradigm generalizes well to various noise types, while also substantially reducing training time. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a general dataset not designed for lipreading. Our dataset was composed of weekly addresses of Barack Obama.

本研究使用基于视听神经网络的视觉语音增强方法，在包含有目标演讲者语音的视频背景噪音情况下，通过口型运动提高演讲者语音的清晰度，在嘈杂的环境中实现了语音增强和噪音降低效果，相较于先前的视听方法在两个公共的口形读音数据集上表现更好，同时也是第一个在面向非口形读音的数据集（如巴拉克·奥巴马每周的演讲）上进行的示例研究。