Existing language models (LMs) predict tokens with a softmax over a finite
vocabulary, which can make it difficult to predict rare tokens or phrases. We
introduce NPM, the first nonparametric masked language model that replaces this
softmax with a nonparametric distribution over every phrase in a reference
corpus. We show that NPM can be efficiently trained with a contrastive
objective and an in-batch approximation to full corpus retrieval. Zero-shot
evaluation on 9 closed-set tasks and 7 open-set tasks demonstrates that NPM
outperforms significantly larger parametric models, either with or without a
retrieve-and-generate approach. It is particularly better on dealing with rare
patterns (word senses or facts), and predicting rare or nearly unseen words
(e.g., non-Latin script). We release the model and code at
this http URL.

本论文介绍了一种基于非参数分布的掩码语言模型，其以参考语料库中的所有短语作为分布依据，通过对比学习和批次近似方法进行高效训练，实现了对罕见词或短语的有效预测和处理，与其他参数化模型比较表明，该模型在 9 项封闭任务和 7 项开放任务中的表现表明了优越性。