Dense passage retrieval (DPR) is the first step in the retrieval augmented
generation (RAG) paradigm for improving the performance of large language
models (LLM). DPR fine-tunes pre-trained networks to enhance the alignment of
the embeddings between queries and relevant textual data. A deeper
understanding of DPR fine-tuning will be required to fundamentally unlock the
full potential of this approach. In this work, we explore DPR-trained models
mechanistically by using a combination of probing, layer activation analysis,
and model editing. Our experiments show that DPR training decentralizes how
knowledge is stored in the network, creating multiple access pathways to the
same information. We also uncover a limitation in this training style: the
internal knowledge of the pre-trained model bounds what the retrieval model can
retrieve. These findings suggest a few possible directions for dense retrieval:
(1) expose the DPR training process to more knowledge so more can be
decentralized, (2) inject facts as decentralized representations, (3) model and
incorporate knowledge uncertainty in the retrieval process, and (4) directly
map internal model knowledge to a knowledge base.

密集路径检索（DPR）是提升大型语言模型（LLM）性能的检索增强生成（RAG）范式中的第一步，本研究通过探测、层激活分析和模型编辑的组合，深入研究 DPR fine-tuning，发现 DPR 训练方式中的去中心化存储及其对检索模型的限制，为密集检索提供了几个可能的方向：（1）将更多知识暴露给 DPR 训练过程以实现更多的去中心化，（2）将事实作为分散表示注入，（3）在检索过程中建模和融入知识的不确定性，以及（4）将内部模型知识直接映射到知识库。