The rise of large language models (LLMs) has transformed the role of
information retrieval (IR) systems in the way to humans accessing information.
Due to the isolated architecture and the limited interaction, existing IR
systems are unable to fully accommodate the shift from directly providing
information to humans to indirectly serving large language models. In this
paper, we propose Self-Retrieval, an end-to-end, LLM-driven information
retrieval architecture that can fully internalize the required abilities of IR
systems into a single LLM and deeply leverage the capabilities of LLMs during
IR process. Specifically, Self-retrieval internalizes the corpus to retrieve
into a LLM via a natural language indexing architecture. Then the entire
retrieval process is redefined as a procedure of document generation and
self-assessment, which can be end-to-end executed using a single large language
model. Experimental results demonstrate that Self-Retrieval not only
significantly outperforms previous retrieval approaches by a large margin, but
also can significantly boost the performance of LLM-driven downstream
applications like retrieval augumented generation.

本文提出了自助检索 (Self-Retrieval) 的一种端到端、以大型语言模型为驱动的信息检索架构，能够充分内化信息检索系统所需的能力到一个单一的大型语言模型，深度利用语言模型在信息检索过程中的能力。实验结果表明，自助检索不仅在很大程度上优于以前的检索方法，还能显著提升以大型语言模型为驱动的下游应用，例如检索增强生成。