Despite their tremendous successes, most large language models do not have
any long-term memory mechanisms, which restricts their applications. Overcoming
this limitation would not only require changes to the typical transformer
architectures or training procedures, but also a dataset on which these new
models could be trained and evaluated. We argue that existing resources lack a
few key properties, and that at present, there are no naturalistic datasets of
sufficient scale to train (and not only evaluate) long-term memory language
models. We then present our solution that capitalizes on the advances in
short-term memory language models to create such a dataset. Using GPT 3.5, we
summarized each scene in 1500 hand-curated books from Project Gutenberg, which
resulted in approximately 150 scene-level summaries per book. We then created a
number of reading comprehension questions based on these summaries, including
three types of multiple-choice scene recognition questions, as well as
free-form narrative reconstruction questions. Each book is thus associated with
more than 500 reading comprehension questions. Crucially, most questions have a
known ``retention demand'', indicating how long-term of a memory is needed to
answer it, which should aid long-term memory performance evaluation. We
validate our data in three small-scale experiments: one with human labelers,
and two with existing language models. We show that our questions 1) adequately
represent the source material 2) can be used to diagnose the model's memory
capacity 3) are not trivial for modern language models even when the memory
demand does not exceed those models' context lengths. Lastly, we provide our
code which can be used to further expand the dataset in an automated manner.

提出了一种通过 GPT 3.5 从 Project Gutenberg 中的书籍总结场景并创建阅读理解问题的方法，用于创建可以训练和评估长期记忆语言模型的数据集，并经过人工和模型的实验验证，证明该数据集可以诊断模型的记忆容量，并且不是现代语言模型的简单问题。

叙述 XL: 用于长期记忆模型的大规模数据集

Narrative XL: A Large-scale Dataset For Long-Term Memory Models

For a natural language understanding benchmark to be useful in research, it
has to consist of examples that are diverse and difficult enough to
discriminate among current and near-future state-of-the-art systems. However,
we do not yet know how best to select text sources to collect a variety of
challenging examples. In this study, we crowdsource multiple-choice reading
comprehension questions for passages taken from seven qualitatively distinct
sources, analyzing what attributes of passages contribute to the difficulty and
question types of the collected examples. To our surprise, we find that passage
source, length, and readability measures do not significantly affect question
difficulty. Through our manual annotation of seven reasoning types, we observe
several trends between passage sources and reasoning types, e.g., logical
reasoning is more often required in questions written for technical passages.
These results suggest that when creating a new benchmark dataset, selecting a
diverse set of passages can help ensure a diverse range of question types, but
that passage difficulty need not be a priority.

通过众包多选阅读理解问题，分析来自七个质量不同的来源的段落的属性如何影响问题难度及类型，发现段落的来源、长度和可读性等因素并不影响问题的难度，而逻辑推理更常常出现在针对技术文章的问题中。因此，在创建新的基准数据集时，选择多样性的段落可以确保问题类型的多样性，而段落的难度则不是最重要的。