Retrieval-Augmented Language Modeling (RALM) by integrating large language
models (LLM) with relevant documents from an external corpus is a proven method
for enabling the LLM to generate information beyond the scope of its
pre-training corpus. Previous work using utilizing retrieved content by simply
prepending retrieved contents to the input poses a high runtime issue, which
degrades the inference efficiency of the LLMs because they fail to use the
Key-Value (KV) cache efficiently. In this paper, we propose \textsc{FlashBack},
a modular RALM designed to improve the inference efficiency of RALM with
appending context pattern while maintaining decent performance after specific
fine-tuning without heavily destruct the knowledge integrity of the LLM.
\textsc{FlashBack} appends retrieved documents at the end of the context for
efficiently utilizing the KV cache instead of prepending them. Our experiment
shows that the inference speed of \textsc{FlashBack} is up to $4\times$ faster
than the prepending method on a 7B LLM (Llama 2). Via bypassing unnecessary
re-computation, it demonstrates an advancement by achieving significantly
faster inference speed, and this heightened efficiency will substantially
reduce inferential cost. Our code will be publicly available.

通过将外部语料库的相关文档与大型语言模型（LLM）集成，检索增强语言建模（RALM）是一种已被证明的方法，可以让 LLM 生成超出其预训练语料库范围的信息。本文提出了 FlashBack，一种模块化的 RALM，通过在上下文末尾附加检索到的文档来提高 RALM 的推理效率，同时在特定的微调后保持良好的性能，而不会严重破坏 LLM 的知识完整性。

FlashBack: 长文本推理的高效检索增强语言建模

FlashBack:Efficient Retrieval-Augmented Language Modeling for Long  Context Inference

Retrieval-Augmented Language Modeling (RALM) methods, that condition a
language model (LM) on relevant documents from a grounding corpus during
generation, have been shown to significantly improve language modeling while
also providing a natural source attribution mechanism. Existing RALM approaches
focus on modifying the LM architecture in order to facilitate the incorporation
of external information, significantly complicating deployment. This paper
proposes an under-explored alternative, which we dub In-Context RALM: leaving
the LM architecture unchanged and prepending grounding documents to the input.
We show that in-context RALM which uses off-the-shelf general purpose
retrievers provides surprisingly large LM gains across model sizes and diverse
corpora. We also demonstrate that the document retrieval and ranking mechanism
can be specialized to the RALM setting to further boost performance. We
conclude that in-context RALM has considerable potential to increase the
prevalence of LM grounding, particularly in settings where a pretrained LM must
be used without modification or even via API access. To that end, we make our
code publicly available.

该论文提出了一种名为 In-Context RALM 的新方法，将基础语料库中的相关文档作为输入前缀添加到语言模型中，实现对外部信息的整合而无需改变模型结构，并证实其在各种场景下的性能提升，从而增加了语言模型的普适性。