Recent works in speech recognition rely either on connectionist temporal
classification (CTC) or sequence-to-sequence models for character-level
recognition. CTC assumes conditional independence of individual characters,
whereas attention-based models can provide nonsequential alignments. Therefore,
we could use a CTC loss in combination with an attention-based model in order
to force monotonic alignments and at the same time get rid of the conditional
independence assumption. In this paper, we use the recently proposed hybrid
CTC/attention architecture for audio-visual recognition of speech in-the-wild.
To the best of our knowledge, this is the first time that such a hybrid
architecture architecture is used for audio-visual recognition of speech. We
use the LRS2 database and show that the proposed audio-visual model leads to an
1.3% absolute decrease in word error rate over the audio-only model and
achieves the new state-of-the-art performance on LRS2 database (7% word error
rate). We also observe that the audio-visual model significantly outperforms
the audio-based model (up to 32.9% absolute improvement in word error rate) for
several different types of noise as the signal-to-noise ratio decreases.

本文介绍了一种基于 CTC/attention 混合架构的音频和视频的语音识别模型，并在 LRS2 数据库上表明这种模型识别率比仅音频模型提高了 1.3 个百分点，并且在噪声环境下表现出色， 是当前状态下最新的结果。