Retrieval augmentation can aid language models (LMs) in knowledge-intensive
tasks by supplying them with external information. Prior works on retrieval
augmentation usually jointly fine-tune the retriever and the LM, making them
closely coupled. In this paper, we explore the scheme of generic retrieval
plug-in: the retriever is to assist target LMs that may not be known beforehand
or are unable to be fine-tuned together. To retrieve useful documents for
unseen target LMs, we propose augmentation-adapted retriever (AAR), which
learns LM's preferences obtained from a known source LM. Experiments on the
MMLU and PopQA datasets demonstrate that our AAR trained with a small source LM
is able to significantly improve the zero-shot generalization of larger target
LMs ranging from 250M Flan-T5 to 175B InstructGPT. Further analysis indicates
that the preferences of different LMs overlap, enabling AAR trained with a
single source LM to serve as a generic plug-in for various target LMs. Our code
is open-sourced at this https URL

以通用的文本检索插件为基础，通过增强适应性的文本检索器 (AAR)，为各种语言模型 (large target LMs ranging from 250M Flan-T5 to 175B InstructGPT) 提供外部知识，从而显着提高零样本推理的泛化能力。