Large Language Models (LLMs) are increasingly applied in various real-world
scenarios due to their excellent generalization capabilities and robust
generative abilities. However, they exhibit position bias, also known as "lost
in the middle", a phenomenon that is especially pronounced in long-context
scenarios, which indicates the placement of the key information in different
positions of a prompt can significantly affect accuracy. This paper first
explores the micro-level manifestations of position bias, concluding that
attention weights are a micro-level expression of position bias. It further
identifies that, in addition to position embeddings, causal attention mask also
contributes to position bias by creating position-specific hidden states. Based
on these insights, we propose a method to mitigate position bias by scaling
this positional hidden states. Experiments on the NaturalQuestions
Multi-document QA, KV retrieval, LongBench and timeline reorder tasks, using
various models including RoPE models, context windowextended models, and Alibi
models, demonstrate the effectiveness and generalizability of our approach. Our
method can improve performance by up to 15.2% by modifying just one dimension
of hidden states. Our code is available at this https URL

本文研究了大型语言模型中的位置偏见问题，发现注意力权重和因果性注意掩码是位置偏见的微观表现，提出了通过调整位置隐状态来减轻位置偏见的方法，并通过在多个任务上进行实验验证了该方法的有效性和普适性。