This paper introduces Neurocache, an approach to extend the effective context
size of large language models (LLMs) using an external vector cache to store
its past states. Like recent vector retrieval approaches, Neurocache uses an
efficient k-nearest-neighbor (kNN) algorithm to retrieve relevant past states
and incorporate them into the attention process. Neurocache improves upon
previous methods by (1) storing compressed states, which reduces cache size;
(2) performing a single retrieval operation per token which increases inference
speed; and (3) extending the retrieval window to neighboring states, which
improves both language modeling and downstream task accuracy. Our experiments
show the effectiveness of Neurocache both for models trained from scratch and
for pre-trained models such as Llama2-7B and Mistral-7B when enhanced with the
cache mechanism. We also compare Neurocache with text retrieval methods and
show improvements in single-document question-answering and few-shot learning
tasks. We made the source code available under:
this https URL

介绍了一种名为 Neurocache 的方法，它使用外部向量缓存扩展了大型语言模型的有效上下文大小，通过存储过去的状态并利用高效的 k 最近邻算法检索相关的过去状态并将其纳入注意力过程中，进而提高语言建模和下游任务准确性。