Large language models (LLMs) augmented with retrieval exhibit robust
performance and extensive versatility by incorporating external contexts.
However, the input length grows linearly in the number of retrieved documents,
causing a dramatic increase in latency. In this paper, we propose a novel
paradigm named Sparse RAG, which seeks to cut computation costs through
sparsity. Specifically, Sparse RAG encodes retrieved documents in parallel,
which eliminates latency introduced by long-range attention of retrieved
documents. Then, LLMs selectively decode the output by only attending to highly
relevant caches auto-regressively, which are chosen via prompting LLMs with
special control tokens. It is notable that Sparse RAG combines the assessment
of each individual document and the generation of the response into a single
process. The designed sparse mechanism in a RAG system can facilitate the
reduction of the number of documents loaded during decoding for accelerating
the inference of the RAG system. Additionally, filtering out undesirable
contexts enhances the model's focus on relevant context, inherently improving
its generation quality. Evaluation results of two datasets show that Sparse RAG
can strike an optimal balance between generation quality and computational
efficiency, demonstrating its generalizability across both short- and long-form
generation tasks.

通过引入稀疏技术，Sparse RAG 提出了一种新颖的范式，在提高生成质量的同时减少计算成本，通过并行编码检索文档并选择性地解码输出，既降低了延迟，又提升了模型的焦点和生成质量。