Retrieval is a widely adopted approach for improving language models
leveraging external information. As the field moves towards multi-modal large
language models, it is important to extend the pure text based methods to
incorporate other modalities in retrieval as well for applications across the
wide spectrum of machine learning tasks and data types. In this work, we
propose multi-modal retrieval with two approaches: kNN-LM and cross-attention
techniques. We demonstrate the effectiveness of our retrieval approaches
empirically by applying them to automatic speech recognition tasks with access
to external information. Under this setting, we show that speech-based
multi-modal retrieval outperforms text based retrieval, and yields up to 50 %
improvement in word error rate over the multi-modal language model baseline.
Furthermore, we achieve state-of-the-art recognition results on the
Spoken-Squad question answering dataset.

提出 kNN-LM 和跨注意力技术两种方法的多模态检索，证明了其在语音识别任务上优于基于文本的检索方法，对多模态语言模型基准线具有最高水平的识别结果。