Fine-tuning a language model on a new domain is standard practice for domain adaptation. However, it can be infeasible when it comes to modern large-scale language models such as GPT-3, which can only be accessed through APIs, making it difficult to access the internal parameters of the model. In this paper, we propose $k$NN-Adapter, a method to effectively adapt these black-box large language models (LLMs) to a new domain. The $k$NN-Adapter builds on top of the retrieval-augmented language model, and adaptively learns to interpolate the output of the language model with retrieval results from a datastore consisting of the target domain data. Our experiments on four different domains demonstrate that $k$NN-Adapter significantly improves perplexity, and works particularly well in settings with limited access to LLMs. Additionally, we show that $k$NN-Adapter is more effective than fine-tuning when the amount of training data is limited. We also release a dataset to encourage further study.

本文介绍了$k$NN-Adapter方法，该方法是基于检索增强语言模型，通过将语言模型的输出和目标域数据存储中的检索结果进行自适应加权插值，有效地适应黑盒大型语言模型到新的领域。实验结果表明，$k$NN-Adapter方法显著提高了语言模型的困惑度，在训练数据有限的情况下比微调更有效。

kNN-Adapter:黑盒语言模型的高效领域自适应