Despite their tremendous successes, most large language models do not have
any long-term memory mechanisms, which restricts their applications. Overcoming
this limitation would not only require changes to the typical transformer
architectures or training procedures, but also a dataset on which these new
models could be trained and evaluated. We argue that existing resources lack a
few key properties, and that at present, there are no naturalistic datasets of
sufficient scale to train (and not only evaluate) long-term memory language
models. We then present our solution that capitalizes on the advances in
short-term memory language models to create such a dataset. Using GPT 3.5, we
summarized each scene in 1500 hand-curated books from Project Gutenberg, which
resulted in approximately 150 scene-level summaries per book. We then created a
number of reading comprehension questions based on these summaries, including
three types of multiple-choice scene recognition questions, as well as
free-form narrative reconstruction questions. Each book is thus associated with
more than 500 reading comprehension questions. Crucially, most questions have a
known ``retention demand'', indicating how long-term of a memory is needed to
answer it, which should aid long-term memory performance evaluation. We
validate our data in three small-scale experiments: one with human labelers,
and two with existing language models. We show that our questions 1) adequately
represent the source material 2) can be used to diagnose the model's memory
capacity 3) are not trivial for modern language models even when the memory
demand does not exceed those models' context lengths. Lastly, we provide our
code which can be used to further expand the dataset in an automated manner.

提出了一种通过 GPT 3.5 从 Project Gutenberg 中的书籍总结场景并创建阅读理解问题的方法，用于创建可以训练和评估长期记忆语言模型的数据集，并经过人工和模型的实验验证，证明该数据集可以诊断模型的记忆容量，并且不是现代语言模型的简单问题。