We explore the use of long-context capabilities in large language models to
create synthetic reading comprehension data from entire books. Previous efforts
to construct such datasets relied on crowd-sourcing, but the emergence of
transformers with a context size of 1 million or more tokens now enables
entirely automatic approaches. Our objective is to test the capabilities of
LLMs to analyze, understand, and reason over problems that require a detailed
comprehension of long spans of text, such as questions involving character
arcs, broader themes, or the consequences of early actions later in the story.
We propose a holistic pipeline for automatic data generation including question
generation, answering, and model scoring using an ``Evaluator''. We find that a
relative approach, comparing answers between models in a pairwise fashion and
ranking with a Bradley-Terry model, provides a more consistent and
differentiating scoring mechanism than an absolute scorer that rates answers
individually. We also show that LLMs from different model families produce
moderate agreement in their ratings. We ground our approach using the manually
curated NarrativeQA dataset, where our evaluator shows excellent agreement with
human judgement and even finds errors in the dataset. Using our automatic
evaluation approach, we show that using an entire book as context produces
superior reading comprehension performance compared to baseline no-context
(parametric knowledge only) and retrieval-based approaches.

通过使用大型语言模型的长期上下文能力，我们利用整本书籍创建合成阅读理解数据。我们提出了一个全面的自动化数据生成流程，包括问题生成、回答和模型评分。我们还发现将答案在模型之间进行相对比较，并使用 Bradley-Terry 模型进行排名，与绝对评分器相比，提供了一种更一致和区分性更强的评分机制。同时，我们展示了不同模型族的大型语言模型在评分方面的适度一致性。通过人工策划的 NarrativeQA 数据集验证了我们的方法，并且我们的评估器与人类判断达到了很好的一致性，甚至在数据集中发现了错误。使用我们的自动化评估方法，我们展示了与无上下文（参数化知识）和基于检索的方法相比，使用整本书籍作为上下文可以获得更优秀的阅读理解性能。