While language Models store a massive amount of world knowledge implicitly in
their parameters, even very large models often fail to encode information about
rare entities and events, while incurring huge computational costs. Recently,
retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated
world knowledge into language generation by leveraging an external
non-parametric index and have demonstrated impressive performance with
constrained model sizes. However, these methods are restricted to retrieving
only textual knowledge, neglecting the ubiquitous amount of knowledge in other
modalities like images -- much of which contains information not covered by any
text. To address this limitation, we propose the first Multimodal
Retrieval-Augmented Transformer (MuRAG), which accesses an external
non-parametric multimodal memory to augment language generation. MuRAG is
pre-trained with a mixture of large-scale image-text and text-only corpora
using a joint contrastive and generative loss. We perform experiments on two
different datasets that require retrieving and reasoning over both images and
text to answer a given query: WebQA, and MultimodalQA. Our results show that
MuRAG achieves state-of-the-art accuracy, outperforming existing models by
10-20\% absolute on both datasets and under both distractor and full-wiki
settings.

MuRAG 是第一个多模态检索增强变压器，利用外部非参数多模态存储器来增强语言生成，并在 WebQA 和 MultimodalQA 两个数据集上实现了最新的准确性，优于现有模型 10-20％绝对值。

MuRAG: 图像和文本开放式问答的多模式检索增强生成器

MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

Recent work on audio-visual navigation assumes a constantly-sounding target
and restricts the role of audio to signaling the target's position. We
introduce semantic audio-visual navigation, where objects in the environment
make sounds consistent with their semantic meaning (e.g., toilet flushing, door
creaking) and acoustic events are sporadic or short in duration. We propose a
transformer-based model to tackle this new semantic AudioGoal task,
incorporating an inferred goal descriptor that captures both spatial and
semantic properties of the target. Our model's persistent multimodal memory
enables it to reach the goal even long after the acoustic event stops. In
support of the new task, we also expand the SoundSpaces audio simulations to
provide semantically grounded sounds for an array of objects in Matterport3D.
Our method strongly outperforms existing audio-visual navigation methods by
learning to associate semantic, acoustic, and visual cues.

介绍了语义音频可视化导航的概念，提出了基于 Transformer 的模型用于处理语义 AudioGoal 任务，拥有持久的多模态记忆，通过学习语义、声音和视觉提示的关联性，可实现在声音事件停止后，仍能到达目标的导航能力。