Fine-tuning a language model on a new domain is standard practice for domain
adaptation. However, it can be infeasible when it comes to modern large-scale
language models such as GPT-3, which can only be accessed through APIs, making
it difficult to access the internal parameters of the model. In this paper, we
propose $k$NN-Adapter, a method to effectively adapt these black-box large
language models (LLMs) to a new domain. The $k$NN-Adapter builds on top of the
retrieval-augmented language model, and adaptively learns to interpolate the
output of the language model with retrieval results from a datastore consisting
of the target domain data. Our experiments on four different domains
demonstrate that $k$NN-Adapter significantly improves perplexity, and works
particularly well in settings with limited access to LLMs. Additionally, we
show that $k$NN-Adapter is more effective than fine-tuning when the amount of
training data is limited. We also release a dataset to encourage further study.

本文介绍了 $k$NN-Adapter 方法，该方法是基于检索增强语言模型，通过将语言模型的输出和目标域数据存储中的检索结果进行自适应加权插值，有效地适应黑盒大型语言模型到新的领域。实验结果表明，$k$NN-Adapter 方法显著提高了语言模型的困惑度，在训练数据有限的情况下比微调更有效。