Large language models (LLMs), even when specifically trained to process long
input contexts, struggle to capture relevant information located in the middle
of their input. This phenomenon has been known as the lost-in-the-middle
problem. In this work, we make three contributions. First, we set out to
understand the factors that cause this phenomenon. In doing so, we establish a
connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs
exhibit a U-shaped attention bias where the tokens at the beginning and at the
end of its input receive higher attention, regardless of their relevance.
Second, we mitigate this positional bias through a calibration mechanism,
found-in-the-middle, that allows the model to attend to contexts faithfully
according to their relevance, even though when they are in the middle. Third,
we show found-in-the-middle not only achieves better performance in locating
relevant information within a long context, but also eventually leads to
improved retrieval-augmented generation (RAG) performance across various tasks,
outperforming existing methods by up to 15 percentage points. These findings
open up future directions in understanding LLM attention bias and its potential
consequences.

长文本中，大语言模型存在失效过中间信息的问题，我们通过研究发现这一现象与其内在的注意力偏差有关，而通过中间定位机制可以解决这一问题，并在检索增强生成任务中表现出更好的性能。

位于中间：校准位置注意偏差以改进长上下文利用

Found in the Middle: Calibrating Positional Attention Bias Improves Long  Context Utilization

In computer vision, the performance of deep neural networks (DNNs) is highly
related to the feature extraction ability, i.e., the ability to recognize and
focus on key pixel regions in an image. However, in this paper, we
quantitatively and statistically illustrate that DNNs have a serious attention
bias problem on many samples from some popular datasets: (1) Position bias:
DNNs fully focus on label-independent regions; (2) Range bias: The focused
regions from DNN are not completely contained in the ideal region. Moreover, we
find that the existing self-attention modules can alleviate these biases to a
certain extent, but the biases are still non-negligible. To further mitigate
them, we propose a lightweight sub-attention strategy (LSAS), which utilizes
high-order sub-attention modules to improve the original self-attention
modules. The effectiveness of LSAS is demonstrated by extensive experiments on
widely-used benchmark datasets and popular attention networks. We release our
code to help other researchers to reproduce the results of
LSAS~\footnote{this https URL}.

本文研究计算机视觉中深度神经网络由于其特征提取能力，会在图像中关注一些关键的像素区域，但我们通过量化和统计表明，DNN 存在严重的注意偏差问题；通过现有的自注意机制有一定的缓解，但仍然存在偏差，因此我们提出了一个轻量级的 LSAS 策略，通过高阶子注意模块来改进原始的自注意模块，并通过实验证明其有效性。