Retrieval-enhanced language models (LMs), which condition their predictions
on text retrieved from large external datastores, have recently shown
significant perplexity improvements compared to standard LMs. One such
approach, the $k$NN-LM, interpolates any existing LM's predictions with the
output of a $k$-nearest neighbors model and requires no additional training. In
this paper, we explore the importance of lexical and semantic matching in the
context of items retrieved by $k$NN-LM. We find two trends: (1) the presence of
large overlapping $n$-grams between the datastore and evaluation set plays an
important factor in strong performance, even when the datastore is derived from
the training data; and (2) the $k$NN-LM is most beneficial when retrieved items
have high semantic similarity with the query. Based on our analysis, we define
a new formulation of the $k$NN-LM that uses retrieval quality to assign the
interpolation coefficient. We empirically measure the effectiveness of our
approach on two English language modeling datasets, Wikitext-103 and PG-19. Our
re-formulation of the $k$NN-LM is beneficial in both cases, and leads to nearly
4% improvement in perplexity on the Wikitext-103 test set.

研究了基于检索增强的语言模型中 $k$NN-LM 中检索文本的词汇和语义匹配对于性能的影响，并通过使用检索结果质量确定插值系数的新表述，成功地提高了英文语言建模数据集 Wikitext-103 和 PG-19 中的困惑度近 4%。