While numerous architectures for long-range language models (LRLMs) have
recently been proposed, a meaningful evaluation of their discourse-level
language understanding capabilities has not yet followed. To this end, we
introduce ChapterBreak, a challenge dataset that provides an LRLM with a long
segment from a narrative that ends at a chapter boundary and asks it to
distinguish the beginning of the ground-truth next chapter from a set of
negative segments sampled from the same narrative. A fine-grained human
annotation reveals that our dataset contains many complex types of chapter
transitions (e.g., parallel narratives, cliffhanger endings) that require
processing global context to comprehend. Experiments on ChapterBreak show that
existing LRLMs fail to effectively leverage long-range context, substantially
underperforming a segment-level model trained directly for this task. We
publicly release our ChapterBreak dataset to spur more principled future
research into LRLMs.

本文介绍 ChapterBreak 数据集，用于评估长距离语言模型（LRLMs）理解语言的能力。实验结果表明，现有的 LRLMs 在处理全局上下文方面表现不佳，相比直接训练用于此任务的分段级别模型，性能显著下降。