Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.

通过在推理过程中引入大规模数据存储，检索增强的语言模型（retrieval-augmented LMs）可以更可靠、适应性更强且更具可追溯性，然而目前在超越知识密集型任务（如问答）以外的领域，检索增强的语言模型尚未被广泛采用，这需要重新考虑数据存储和检索器、改进检索器和语言模型组件之间的交互以及在高效训练和推理方面进行大规模投资。

可靠、适应性强、可归因的检索式语言模型