We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM)
by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The
nearest neighbors are computed according to distance in the pre-trained LM
embedding space, and can be drawn from any text collection, including the
original LM training data. Applying this augmentation to a strong Wikitext-103
LM, with neighbors drawn from the original training set, our $k$NN-LM achieves
a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no
additional training. We also show that this approach has implications for
efficiently scaling up to larger training sets and allows for effective domain
adaptation, by simply varying the nearest neighbor datastore, again without
further training. Qualitatively, the model is particularly helpful in
predicting rare patterns, such as factual knowledge. Together, these results
strongly suggest that learning similarity between sequences of text is easier
than predicting the next word, and that nearest neighbor search is an effective
approach for language modeling in the long tail.

引入了 $k$NN-LMs，该模型将预训练的神经语言模型与 $k$ 最近邻居模型线性插值。使用此方法在一个强大的 Wikitext-103 LM 中，我们实现了一个新的最先进的困惑度为 15.79，这是一个 2.9 点的提高而无需额外的训练。此外，作者还展示了这种方法在有效地扩展到更大的训练数据和实现领域自适应方面的作用，并认为最近邻搜索是在长尾系统的语言建模中一种有效的方法。