While language Models store a massive amount of world knowledge implicitly in
their parameters, even very large models often fail to encode information about
rare entities and events, while incurring huge computational costs. Recently,
retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated
world knowledge into language generation by leveraging an external
non-parametric index and have demonstrated impressive performance with
constrained model sizes. However, these methods are restricted to retrieving
only textual knowledge, neglecting the ubiquitous amount of knowledge in other
modalities like images -- much of which contains information not covered by any
text. To address this limitation, we propose the first Multimodal
Retrieval-Augmented Transformer (MuRAG), which accesses an external
non-parametric multimodal memory to augment language generation. MuRAG is
pre-trained with a mixture of large-scale image-text and text-only corpora
using a joint contrastive and generative loss. We perform experiments on two
different datasets that require retrieving and reasoning over both images and
text to answer a given query: WebQA, and MultimodalQA. Our results show that
MuRAG achieves state-of-the-art accuracy, outperforming existing models by
10-20\% absolute on both datasets and under both distractor and full-wiki
settings.

MuRAG 是第一个多模态检索增强变压器，利用外部非参数多模态存储器来增强语言生成，并在 WebQA 和 MultimodalQA 两个数据集上实现了最新的准确性，优于现有模型 10-20％绝对值。