While automatic speech recognition (ASR) systems degrade significantly in
noisy environments, audio-visual speech recognition (AVSR) systems aim to
complement the audio stream with noise-invariant visual cues and improve the
system's robustness. However, current studies mainly focus on fusing the
well-learned modality features, like the output of modality-specific encoders,
without considering the contextual relationship during the modality feature
learning. In this study, we propose a multi-layer cross-attention fusion based
AVSR (MLCA-AVSR) approach that promotes representation learning of each
modality by fusing them at different levels of audio/visual encoders.
Experimental results on the MISP2022-AVSR Challenge dataset show the efficacy
of our proposed system, achieving a concatenated minimum permutation character
error rate (cpCER) of 30.57% on the Eval set and yielding up to 3.17% relative
improvement compared with our previous system which ranked the second place in
the challenge. Following the fusion of multiple systems, our proposed approach
surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on
this dataset.

提出了一种多层交叉注意力融合的视听语音识别方法，通过在不同的音频 / 视觉编码器层级融合各种模态，实现了每种模态的表示学习，实验结果表明该方法在 MISP2022-AVSR 挑战数据集上达到了新的拼接最小排列字符错误率 (cpCER) 为 30.57% 的性能，并相对于前期系统获得了最多 3.17% 的相对改进，同时超过了第一名系统，获得了该数据集上的新的最先进 cpCER 为 29.13%。