Large language models (LLMs), even when specifically trained to process long
input contexts, struggle to capture relevant information located in the middle
of their input. This phenomenon has been known as the lost-in-the-middle
problem. In this work, we make three contributions. First, we set out to
understand the factors that cause this phenomenon. In doing so, we establish a
connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs
exhibit a U-shaped attention bias where the tokens at the beginning and at the
end of its input receive higher attention, regardless of their relevance.
Second, we mitigate this positional bias through a calibration mechanism,
found-in-the-middle, that allows the model to attend to contexts faithfully
according to their relevance, even though when they are in the middle. Third,
we show found-in-the-middle not only achieves better performance in locating
relevant information within a long context, but also eventually leads to
improved retrieval-augmented generation (RAG) performance across various tasks,
outperforming existing methods by up to 15 percentage points. These findings
open up future directions in understanding LLM attention bias and its potential
consequences.

长文本中，大语言模型存在失效过中间信息的问题，我们通过研究发现这一现象与其内在的注意力偏差有关，而通过中间定位机制可以解决这一问题，并在检索增强生成任务中表现出更好的性能。

位于中间：校准位置注意偏差以改进长上下文利用

Found in the Middle: Calibrating Positional Attention Bias Improves Long  Context Utilization

Recently, many methods have been developed to extend the context length of
pre-trained large language models (LLMs), but they often require fine-tuning at
the target length ($\gg4K$) and struggle to effectively utilize information
from the middle part of the context. To address these issues, we propose
$\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with
g$\textbf{A}$ussian $\textbf{M}$iddle (CREAM), which interpolates positional
encodings by manipulating position indices. Apart from being simple, CREAM is
training-efficient: it only requires fine-tuning at the pre-trained context
window (eg, Llama 2-4K) and can extend LLMs to a much longer target context
length (eg, 256K). To ensure that the model focuses more on the information in
the middle, we introduce a truncated Gaussian to encourage sampling from the
middle part of the context during fine-tuning, thus alleviating the
``Lost-in-the-Middle'' problem faced by long-context LLMs. Experimental results
show that CREAM successfully extends LLMs to the target length for both Base
and Chat versions of $\texttt{Llama2-7B}$ with ``Never Miss A Beat''. Our code
will be publicly available soon.

提出了一种称为 CREAM 的方法，通过操纵位置索引插值位置编码，从而实现对预训练大型语言模型的上下文长度进行扩展，并解决长上下文模型面临的 “丢失在中间” 问题。

从容不迫：大型语言模型上上下文窗口扩展的高效配方，持续关注 “中间” 增强

Never Miss A Beat: An Efficient Recipe for Context Window Extension of  Large Language Models with Consistent "Middle" Enhancement

Large Language Models (LLMs) have shown promising performance in summary
evaluation tasks, yet they face challenges such as high computational costs and
the Lost-in-the-Middle problem where important information in the middle of
long documents is often overlooked. To address these issues, this paper
introduces a novel approach, Extract-then-Evaluate, which involves extracting
key sentences from a long source document and then evaluating the summary by
prompting LLMs. The results reveal that the proposed method not only
significantly reduces evaluation costs but also exhibits a higher correlation
with human evaluations. Furthermore, we provide practical recommendations for
optimal document length and sentence extraction methods, contributing to the
development of cost-effective yet more accurate methods for LLM-based text
generation evaluation.

该研究提出了一种新方法，即从长文档中提取关键句子，然后通过提示大型语言模型来评估摘要，以解决计算成本高、长文档中的重要信息往往被忽视的问题，研究结果显示该方法不仅显著降低了评估成本，而且与人工评估的相关性更高，此外，我们还提供了关于最佳文档长度和句子提取方法的实用建议，为基于大型语言模型的文本生成评估的成本效益更高且更准确的方法的发展做出了贡献。