Pre-trained language models demonstrate general intelligence and common
sense, but long inputs quickly become a bottleneck for memorizing information
at inference time. We resurface a simple method, Memorizing Transformers (Wu et
al., 2022), that gives the model access to a bank of pre-computed memories. We
show that it is possible to fix many of the shortcomings of the original
method, such as the need for fine-tuning, by critically assessing how
positional encodings should be updated for the keys and values retrieved. This
intuitive method uses the model's own key/query system to select and attend to
the most relevant memories at each generation step, rather than using external
embeddings. We demonstrate the importance of external information being
retrieved in a majority of decoder layers, contrary to previous work. We open
source a new counterfactual long-range retrieval benchmark, and show that
Extended Mind Transformers outperform today's state of the art by 6% on
average.

预训练语言模型在推理时过长的输入会迅速成为记忆信息的瓶颈。本文提出了一种简单的方法，使用记忆转换器给模型提供预先计算的记忆库，通过批判性评估定位编码应如何更新以检索关键信息，而非使用外部嵌入。我们展示了在大多数解码器层中检索外部信息的重要性，并公开了一个新的反事实长距离检索基准。实验证明，扩展心智转换器在平均性能上超过了当今的最先进水平。