Considering the bimodal nature of human speech perception, lips, and teeth
movement has a pivotal role in automatic speech recognition. Benefiting from
the correlated and noise-invariant visual information, audio-visual recognition
systems enhance robustness in multiple scenarios. In previous work,
audio-visual HuBERT appears to be the finest practice incorporating modality
knowledge. This paper outlines a mixed methodology, named conformer enhanced
AV-HuBERT, boosting the AV-HuBERT system's performance a step further. Compared
with baseline AV-HuBERT, our method in the one-phase evaluation of clean and
noisy conditions achieves 7% and 16% relative WER reduction on the English AVSR
benchmark dataset LRS3. Furthermore, we establish a novel 1000h Mandarin AVSR
dataset CSTS. On top of the baseline AV-HuBERT, we exceed the WeNet ASR system
by 14% and 18% relatively on MISP and CMLR by pre-training with this dataset.
The conformer-enhanced AV-HuBERT we proposed brings 7% on MISP and 6% CER
reduction on CMLR, compared with the baseline AV-HuBERT system.

该论文介绍了一种混合方法，名为 conformer-enhanced AV-HuBERT，该方法在语音识别方面的表现进一步提高。该方法基于 AV-HuBERT，在一个清洁和嘈杂环境下分别实现了相对 WER 降低 7％和 16％。除此之外，该论文还提出了一个新的 1000 小时的普通话语音识别数据集 CSTS，通过预先训练，该方法相对于基线 AV-HuBERT 超过了 WeNet ASR 系统。conformer-enhanced AV-HuBERT 相对于基线 AV-HuBERT 系统，在 MISP 和 CMLR 上分别减少了 7％和 6％的 CER。