Retrieval-Augmented Language Modeling (RALM) methods, that condition a
language model (LM) on relevant documents from a grounding corpus during
generation, have been shown to significantly improve language modeling while
also providing a natural source attribution mechanism. Existing RALM approaches
focus on modifying the LM architecture in order to facilitate the incorporation
of external information, significantly complicating deployment. This paper
proposes an under-explored alternative, which we dub In-Context RALM: leaving
the LM architecture unchanged and prepending grounding documents to the input.
We show that in-context RALM which uses off-the-shelf general purpose
retrievers provides surprisingly large LM gains across model sizes and diverse
corpora. We also demonstrate that the document retrieval and ranking mechanism
can be specialized to the RALM setting to further boost performance. We
conclude that in-context RALM has considerable potential to increase the
prevalence of LM grounding, particularly in settings where a pretrained LM must
be used without modification or even via API access. To that end, we make our
code publicly available.

该论文提出了一种名为 In-Context RALM 的新方法，将基础语料库中的相关文档作为输入前缀添加到语言模型中，实现对外部信息的整合而无需改变模型结构，并证实其在各种场景下的性能提升，从而增加了语言模型的普适性。