Long-context modeling capabilities have garnered widespread attention,
leading to the emergence of Large Language Models (LLMs) with ultra-context
windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually
catching up. However, existing benchmarks employ irrelevant noise texts to
artificially extend the length of test cases, diverging from the real-world
scenarios of long-context applications. To bridge this gap, we propose a novel
long-context benchmark, Loong, aligning with realistic scenarios through
extended multi-document question answering (QA). Unlike typical document QA, in
Loong's test cases, each document is relevant to the final answer, ignoring any
document will lead to the failure of the answer. Furthermore, Loong introduces
four types of tasks with a range of context lengths: Spotlight Locating,
Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic
and comprehensive evaluation of long-context understanding. Extensive
experiments indicate that existing long-context language models still exhibit
considerable potential for enhancement. Retrieval augmented generation (RAG)
achieves poor performance, demonstrating that Loong can reliably assess the
model's long-context modeling capabilities.

提出了一个新的长上下文基准测试 Loong，通过扩展的多文档问题回答来实现与现实场景的对齐，来评估模型的长上下文建模能力。

不留下任何文件：扩展多文档问答中的长上下文语言模型基准测试

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended  Multi-Doc QA

Large language models (LLMs) excel in zero-shot abstractive summarization
tasks, delivering fluent and pertinent summaries. Recent advancements have
extended their capabilities to handle long-input contexts, surpassing token
limits of 32k or more. However, in the realm of multi-document question
answering, language models exhibit uneven utilization of their input context.
They tend to favor the initial and final segments, resulting in a U-shaped
performance pattern concerning where the answer is located within the input.
This bias raises concerns, particularly in summarization tasks where crucial
content may be dispersed throughout the source document(s). This paper presents
a comprehensive investigation encompassing 10 datasets, 4 LLMs, and 5
evaluation metrics to analyze how these models leverage their input for
abstractive summarization. Our findings reveal a pronounced bias towards the
introductory content (and to a lesser extent, the final content), posing
challenges for LLM performance across a range of diverse summarization
benchmarks.

对于大型语言模型，在抽象摘要任务中表现出色，但在多文档问答中存在输入上下文偏差，导致摘要内容分散，影响性能。本文通过实证研究揭示了这种偏差对于大型语言模型在不同摘要评估上的挑战。

关于大型语言模型中的位置偏差的总结化问题

On Position Bias in Summarization with Large Language Models

The 'pre-train, prompt, predict' paradigm of large language models (LLMs) has
achieved remarkable success in open-domain question answering (OD-QA). However,
few works explore this paradigm in the scenario of multi-document question
answering (MD-QA), a task demanding a thorough understanding of the logical
associations among the contents and structures of different documents. To fill
this crucial gap, we propose a Knowledge Graph Prompting (KGP) method to
formulate the right context in prompting LLMs for MD-QA, which consists of a
graph construction module and a graph traversal module. For graph construction,
we create a knowledge graph (KG) over multiple documents with nodes symbolizing
passages or document structures (e.g., pages/tables), and edges denoting the
semantic/lexical similarity between passages or intra-document structural
relations. For graph traversal, we design an LM-guided graph traverser that
navigates across nodes and gathers supporting passages assisting LLMs in MD-QA.
The constructed graph serves as the global ruler that regulates the
transitional space among passages and reduces retrieval latency. Concurrently,
the LM-guided traverser acts as a local navigator that gathers pertinent
context to progressively approach the question and guarantee retrieval quality.
Extensive experiments underscore the efficacy of KGP for MD-QA, signifying the
potential of leveraging graphs in enhancing the prompt design for LLMs. Our
code is at this https URL

在多文档问答（MD-QA）方案中，使用知识图谱引导大型语言模型进行提示设计的带有图构造和遍历模块的知识图谱提示（KGP）方法，在提取支持材料和减少检索延迟方面取得了显著效果，并显示出利用图形增强大型语言模型中的提问设计的潜力。