Humans are adept at leveraging visual cues from lip movements for recognizing
speech in adverse listening conditions. Audio-Visual Speech Recognition (AVSR)
models follow similar approach to achieve robust speech recognition in noisy
conditions. In this work, we present a multilingual AVSR model incorporating
several enhancements to improve performance and audio noise robustness.
Notably, we adapt the recently proposed Fast Conformer model to process both
audio and visual modalities using a novel hybrid CTC/RNN-T architecture. We
increase the amount of audio-visual training data for six distinct languages,
generating automatic transcriptions of unlabelled multilingual datasets
(VoxCeleb2 and AVSpeech). Our proposed model achieves new state-of-the-art
performance on the LRS3 dataset, reaching WER of 0.8%. On the recently
introduced MuAViC benchmark, our model yields an absolute average-WER reduction
of 11.9% in comparison to the original baseline. Finally, we demonstrate the
ability of the proposed model to perform audio-only, visual-only, and
audio-visual speech recognition at test time.

通过引入多语种音视频语音识别模型和快速调整器模型，提高了模型性能和音频噪音稳健性。在多语种数据集上进行训练，达到了领先水平，并在 MuAViC 基准测试中显著减少了平均识别错误率。