The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
a piece of information (the "needle") from long distractor texts (the
"haystack"), has been widely adopted to evaluate long-context language models
(LMs). However, this simple retrieval-based test is indicative of only a
superficial form of long-context understanding. To provide a more comprehensive
evaluation of long-context LMs, we create a new synthetic benchmark RULER with
flexible configurations for customized sequence length and task complexity.
RULER expands upon the vanilla NIAH test to encompass variations with diverse
types and quantities of needles. Moreover, RULER introduces new task categories
multi-hop tracing and aggregation to test behaviors beyond searching from
context. We evaluate ten long-context LMs with 13 representative tasks in
RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
models exhibit large performance drops as the context length increases. While
these models all claim context sizes of 32K tokens or greater, only four models
(GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
at the length of 32K. Our analysis of Yi-34B, which supports context length of
200K, reveals large room for improvement as we increase input length and task
complexity. We open source RULER to spur comprehensive evaluation of
long-context LMs.

评估长上下文语言模型的新综合性基准 RULER，包括类别多跳追踪和聚合测试行为，显示出在上下文长度增加时所提供的任务和性能急剧下降