Visual context has been shown to be useful for automatic speech recognition
(ASR) systems when the speech signal is noisy or corrupted. Previous work,
however, has only demonstrated the utility of visual context in an unrealistic
setting, where a fixed set of words are systematically masked in the audio. In
this paper, we simulate a more realistic masking scenario during model
training, called RandWordMask, where the masking can occur for any word
segment. Our experiments on the Flickr 8K Audio Captions Corpus show that
multimodal ASR can generalize to recover different types of masked words in
this unstructured masking setting. Moreover, our analysis shows that our models
are capable of attending to the visual signal when the audio signal is
corrupted. These results show that multimodal ASR systems can leverage the
visual signal in more generalized noisy scenarios.

本篇论文研究了在嘈杂的情况下，如何通过视觉上下文提升语音识别的准确性，并通过模拟 RandWordMask 掩码模式验证了多模态 ASR 系统在不同掩码模式下的泛化能力，结论显示在一定程度上可提升 ASR 系统的能力。