Audio-based automatic speech recognition (ASR) degrades significantly in
noisy environments and is particularly vulnerable to interfering speech, as the
model cannot determine which speaker to transcribe. Audio-visual speech
recognition (AVSR) systems improve robustness by complementing the audio stream
with the visual information that is invariant to noise and helps the model
focus on the desired speaker. However, previous AVSR work focused solely on the
supervised learning setup; hence the progress was hindered by the amount of
labeled data available. In this work, we present a self-supervised AVSR
framework built upon Audio-Visual HuBERT (AV-HuBERT), a state-of-the-art
audio-visual speech representation learning model. On the largest available
AVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by
~50% (28.0% vs. 14.1%) using less than 10% of labeled data (433hr vs. 30hr) in
the presence of babble noise, while reducing the WER of an audio-based model by
over 75% (25.8% vs. 5.8%) on average.

本文提出了一个基于 AV-HuBERT 模型的自监督音视频言语识别框架，利用 LRS3 数据集的少量标记数据，在噪音干扰的情况下提高了超过 50% 的性能，并且比基于音频的模型将词错误率减少了 75% 以上。