Our objective is an audio-visual model for separating a single speaker from a
mixture of sounds such as other speakers and background noise. Moreover, we
wish to hear the speaker even when the visual cues are temporarily absent due
to occlusion. To this end we introduce a deep audio-visual speech enhancement
network that is able to separate a speaker's voice by conditioning on both the
speaker's lip movements and/or a representation of their voice. The voice
representation can be obtained by either (i) enrollment, or (ii) by
self-enrollment -- learning the representation on-the-fly given sufficient
unobstructed visual input. The model is trained by blending audios, and by
introducing artificial occlusions around the mouth region that prevent the
visual modality from dominating. The method is speaker-independent, and we
demonstrate it on real examples of speakers unheard (and unseen) during
training. The method also improves over previous models in particular for cases
of occlusion in the visual modality.

本文提出了一种深度音频 - 视觉语音增强网络，该网络可以通过对说话者的嘴唇动作和 / 或声音来分离说话者的声音，在嘴部区域引入人工遮挡并通过混合音频来训练模型，实现说话者独立，且在视觉感知受阻的情况下表现优异的应用。

嘴唇闭合：透过障碍物进行视听言语增强

My lips are concealed: Audio-visual speech enhancement through  obstructions

Recent works in speech recognition rely either on connectionist temporal
classification (CTC) or sequence-to-sequence models for character-level
recognition. CTC assumes conditional independence of individual characters,
whereas attention-based models can provide nonsequential alignments. Therefore,
we could use a CTC loss in combination with an attention-based model in order
to force monotonic alignments and at the same time get rid of the conditional
independence assumption. In this paper, we use the recently proposed hybrid
CTC/attention architecture for audio-visual recognition of speech in-the-wild.
To the best of our knowledge, this is the first time that such a hybrid
architecture architecture is used for audio-visual recognition of speech. We
use the LRS2 database and show that the proposed audio-visual model leads to an
1.3% absolute decrease in word error rate over the audio-only model and
achieves the new state-of-the-art performance on LRS2 database (7% word error
rate). We also observe that the audio-visual model significantly outperforms
the audio-based model (up to 32.9% absolute improvement in word error rate) for
several different types of noise as the signal-to-noise ratio decreases.

本文介绍了一种基于 CTC/attention 混合架构的音频和视频的语音识别模型，并在 LRS2 数据库上表明这种模型识别率比仅音频模型提高了 1.3 个百分点，并且在噪声环境下表现出色， 是当前状态下最新的结果。

基于混合 CTC / 注意力架构的音视频语音识别

Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture

We present a joint audio-visual model for isolating a single speech signal
from a mixture of sounds such as other speakers and background noise. Solving
this task using only audio as input is extremely challenging and does not
provide an association of the separated speech signals with speakers in the
video. In this paper, we present a deep network-based model that incorporates
both visual and auditory signals to solve this task. The visual features are
used to "focus" the audio on desired speakers in a scene and to improve the
speech separation quality. To train our joint audio-visual model, we introduce
AVSpeech, a new dataset comprised of thousands of hours of video segments from
the Web. We demonstrate the applicability of our method to classic speech
separation tasks, as well as real-world scenarios involving heated interviews,
noisy bars, and screaming children, only requiring the user to specify the face
of the person in the video whose speech they want to isolate. Our method shows
clear advantage over state-of-the-art audio-only speech separation in cases of
mixed speech. In addition, our model, which is speaker-independent (trained
once, applicable to any speaker), produces better results than recent
audio-visual speech separation methods that are speaker-dependent (require
training a separate model for each speaker of interest).

我们提出了一种联合音频 - 视觉模型，用于从包括其它说话者和背景噪声在内的混合声音中分离单个语音信号，并且利用深度学习和 AVSpeech 数据集，实现了人脸指定后的语音分离任务，相较于仅使用音频的方法，在多说话人的情况下表现出更好的效果。