Recently, continuous cache models were proposed as extensions to recurrent
neural network language models, to adapt their predictions to local changes in
the data distribution. These models only capture the local context, of up to a
few thousands tokens. In this paper, we propose an extension of continuous
cache models, which can scale to larger contexts. In particular, we use a large
scale non-parametric memory component that stores all the hidden activations
seen in the past. We leverage recent advances in approximate nearest neighbor
search and quantization algorithms to store millions of representations while
searching them efficiently. We conduct extensive experiments showing that our
approach significantly improves the perplexity of pre-trained language models
on new distributions, and can scale efficiently to much larger contexts than
previously proposed local cache models.

提出了一种扩展连续缓存模型的方法，通过使用大规模的非参数内存组件，存储过去观察到的所有隐藏激活，并利用近似最近邻搜索和量化算法，高效地存储和搜索数百万的表示，显著提高了预训练语言模型在新分布上的困惑度，并且可以有效地扩展到比以前提出的本地缓存模型更大的上下文范围。