Long-context language models (LCLMs) have the potential to revolutionize our
approach to tasks traditionally reliant on external tools like retrieval
systems or databases. Leveraging LCLMs' ability to natively ingest and process
entire corpora of information offers numerous advantages. It enhances
user-friendliness by eliminating the need for specialized knowledge of tools,
provides robust end-to-end modeling that minimizes cascading errors in complex
pipelines, and allows for the application of sophisticated prompting techniques
across the entire system. To assess this paradigm shift, we introduce LOFT, a
benchmark of real-world tasks requiring context up to millions of tokens
designed to evaluate LCLMs' performance on in-context retrieval and reasoning.
Our findings reveal LCLMs' surprising ability to rival state-of-the-art
retrieval and RAG systems, despite never having been explicitly trained for
these tasks. However, LCLMs still face challenges in areas like compositional
reasoning that are required in SQL-like tasks. Notably, prompting strategies
significantly influence performance, emphasizing the need for continued
research as context lengths grow. Overall, LOFT provides a rigorous testing
ground for LCLMs, showcasing their potential to supplant existing paradigms and
tackle novel tasks as model capabilities scale.

基于长上下文语言模型（LCLMs）的 LOFT 基准评估了 LCLMs 在上下文检索和推理方面的性能，发现它们能与最先进的检索系统和 RAG 系统媲美，但在像 SQL 一样需要组合推理的领域仍存在挑战，提示着需要继续研究随着上下文长度增长时的提示策略的影响。LOFT 为 LCLMs 提供了严格的测试平台，展示了随着模型能力的扩展，它们取代现有范式并解决新任务的潜力。

长上下文语言模型是否能包含检索、RAG、SQL 等功能？

Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
a piece of information (the "needle") from long distractor texts (the
"haystack"), has been widely adopted to evaluate long-context language models
(LMs). However, this simple retrieval-based test is indicative of only a
superficial form of long-context understanding. To provide a more comprehensive
evaluation of long-context LMs, we create a new synthetic benchmark RULER with
flexible configurations for customized sequence length and task complexity.
RULER expands upon the vanilla NIAH test to encompass variations with diverse
types and quantities of needles. Moreover, RULER introduces new task categories
multi-hop tracing and aggregation to test behaviors beyond searching from
context. We evaluate ten long-context LMs with 13 representative tasks in
RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
models exhibit large performance drops as the context length increases. While
these models all claim context sizes of 32K tokens or greater, only four models
(GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
at the length of 32K. Our analysis of Yi-34B, which supports context length of
200K, reveals large room for improvement as we increase input length and task
complexity. We open source RULER to spur comprehensive evaluation of
long-context LMs.

评估长上下文语言模型的新综合性基准 RULER，包括类别多跳追踪和聚合测试行为，显示出在上下文长度增加时所提供的任务和性能急剧下降