Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly cater to long-text comprehension by expanding the input of traditional tasks, which falls short to exhibit the unique characteristics of long-text understanding, including long dependency tasks and longer text length compatible with modern LLMs' context window size. In this paper, we introduce a benchmark for extremely long context understanding with long-range dependencies, XL$^2$Bench, which includes three scenarios: Fiction Reading, Paper Reading, and Law Reading, and four tasks of increasing complexity: Memory Retrieval, Detailed Understanding, Overall Understanding, and Open-ended Generation, covering 27 subtasks in English and Chinese. It has an average length of 100K+ words (English) and 200K+ characters (Chinese). Evaluating six leading LLMs on XL$^2$Bench, we find that their performance significantly lags behind human levels. Moreover, the observed decline in performance across both the original and enhanced datasets underscores the efficacy of our approach to mitigating data contamination.

我们介绍了一个长文本理解的基准测试XL2Bench，其中包含三个场景：小说阅读、论文阅读和法律阅读，以及四个难度递增的任务：记忆检索、细节理解、整体理解和开放式生成，涵盖了27个子任务，用英文和中文表示，平均长度为100K+词（英文）和200K+字符（中文）。我们评估了六个领先的大型语言模型在XL2Bench上的表现，发现它们的性能明显落后于人类水平。此外，我们观察到在原始数据集和增强数据集上的性能下降，凸显了我们缓解数据污染的方法的有效性。

XL$^2$Bench：极长上下文理解与长距离依赖的基准测试