Audio-visual speech recognition (AVSR) provides a promising solution to
ameliorate the noise-robustness of audio-only speech recognition with visual
information. However, most existing efforts still focus on audio modality to
improve robustness considering its dominance in AVSR task, with noise
adaptation techniques such as front-end denoise processing. Though effective,
these methods are usually faced with two practical challenges: 1) lack of
sufficient labeled noisy audio-visual training data in some real-world
scenarios and 2) less optimal model generality to unseen testing noises. In
this work, we investigate the noise-invariant visual modality to strengthen
robustness of AVSR, which can adapt to any testing noises while without
dependence on noisy training data, a.k.a., unsupervised noise adaptation.
Inspired by human perception mechanism, we propose a universal viseme-phoneme
mapping (UniVPM) approach to implement modality transfer, which can restore
clean audio from visual signals to enable speech recognition under any noisy
conditions. Extensive experiments on public benchmarks LRS3 and LRS2 show that
our approach achieves the state-of-the-art under various noisy as well as clean
conditions. In addition, we also outperform previous state-of-the-arts on
visual speech recognition task.

本文提出了一种基于视觉模态的无监督噪声适应方案，使用通用的音位 - 音素映射 (UniVPM) 技术从视觉信号中恢复干净音频从而扩展 AVSR 系统的噪声鲁棒性。在公共基准测试数据集 LRS3 和 LRS2 上进行的广泛实验表明，我们的方法在各种噪声和清晰条件下都达到了最先进水平，此外在视觉语音识别任务上也优于之前的最优水平。