End-to-end Automatic Speech Recognition (ASR) systems based on neural
networks have seen large improvements in recent years. The availability of
large scale hand-labeled datasets and sufficient computing resources made it
possible to train powerful deep neural networks, reaching very low Word Error
Rate (WER) on academic benchmarks. However, despite impressive performance on
clean audio samples, a drop of performance is often observed on noisy speech.
In this work, we propose to improve the noise robustness of the recently
proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based
architecture by processing both audio and visual modalities. We improve
previous lip reading methods using an Efficient Conformer back-end on top of a
ResNet-18 visual front-end and by adding intermediate CTC losses between
blocks. We condition intermediate block features on early predictions using
Inter CTC residual modules to relax the conditional independence assumption of
CTC-based models. We also replace the Efficient Conformer grouped attention by
a more efficient and simpler attention mechanism that we call patch attention.
We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip
Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and
visual modalities allows to better recognize speech in the presence of
environmental noise and significantly accelerate training, reaching lower WER
with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC)
model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on
LRS2 and LRS3 test sets. Code and pretrained models are available at
this https URL.

本研究建议使用音频和视觉模态来改善基于 Efficient Conformer Connectionist Temporal Classification 架构的噪声鲁棒性，并在 LRS2 和 LRS3 数据集上进行了实验。结果表明，使用音频和视觉模态可以更好地识别存在环境噪声的语音，并显着加速训练，达到了 2.3％和 1.8％的字错误率。