Audio-visual speech recognition (AVSR) research has gained a great success
recently by improving the noise-robustness of audio-only automatic speech
recognition (ASR) with noise-invariant visual information. However, most
existing AVSR approaches simply fuse the audio and visual features by
concatenation, without explicit interactions to capture the deep correlations
between them, which results in sub-optimal multimodal representations for
downstream speech recognition task. In this paper, we propose a cross-modal
global interaction and local alignment (GILA) approach for AVSR, which captures
the deep audio-visual (A-V) correlations from both global and local
perspectives. Specifically, we design a global interaction model to capture the
A-V complementary relationship on modality level, as well as a local alignment
approach to model the A-V temporal consistency on frame level. Such a holistic
view of cross-modal correlations enable better multimodal representations for
AVSR. Experiments on public benchmarks LRS3 and LRS2 show that our GILA
outperforms the supervised learning state-of-the-art.

本文提出了一种跨模态全局交互和局部对齐 (GILA) 方法，从全局和局部角度捕捉音频 - 视觉 (A-V) 间的深层相关性，用于改善音频 - 视觉语音识别中的多模态表示，实验结果表明我们的方法优于现有的有监督学习方法。