This paper introduces xRAG, an innovative context compression method tailored
for retrieval-augmented generation. xRAG reinterprets document embeddings in
dense retrieval--traditionally used solely for retrieval--as features from the
retrieval modality. By employing a modality fusion methodology, xRAG seamlessly
integrates these embeddings into the language model representation space,
effectively eliminating the need for their textual counterparts and achieving
an extreme compression rate. In xRAG, the only trainable component is the
modality bridge, while both the retriever and the language model remain frozen.
This design choice allows for the reuse of offline-constructed document
embeddings and preserves the plug-and-play nature of retrieval augmentation.
Experimental results demonstrate that xRAG achieves an average improvement of
over 10% across six knowledge-intensive tasks, adaptable to various language
model backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts
configuration. xRAG not only significantly outperforms previous context
compression methods but also matches the performance of uncompressed models on
several datasets, while reducing overall FLOPs by a factor of 3.53. Our work
pioneers new directions in retrieval-augmented generation from the perspective
of multimodality fusion, and we hope it lays the foundation for future
efficient and scalable retrieval-augmented systems

xRAG 是一种针对检索增强生成的创新上下文压缩方法，通过模态融合的方法将文档嵌入重新解释为来自检索模态的特征，并在语言模型表示空间中无缝集成这些嵌入以实现极高的压缩率。实验证明，xRAG 在六个强调知识的任务中达到了平均改进 10% 以上，适用于各种语言模型主干结构，从密集 7B 模型到 8x7B 多专家结构。xRAG 不仅显著超越了以前的上下文压缩方法，而且在减少计算量（FLOPs）的同时，与未压缩模型在多个数据集上保持了相当的性能，为未来有效和可伸缩的检索增强系统奠定了基础。