The context window of large language models has been extended to 128k tokens
or more. However, language models still suffer from position bias and have
difficulty in accessing and using the middle part of the context due to the
lack of attention. We examine the relative position awareness of LLMs and the
feasibility of mitigating disproportional attention through prompting. We
augment the original task instruction with $\texttt{attention instructions}$
that direct language models to allocate more attention towards a selected
segment of the context. We conduct a comprehensive investigation on
multi-document question answering task with both position-based and index-based
instructions. We find that language models do not have relative position
awareness of the context. Nevertheless, they demonstrate the capacity to adapt
attention to a specific segment using matching indexes. Our analysis
contributes to a deeper understanding of position bias in LLMs and provides a
pathway to mitigate this bias by instruction, thus benefiting LLMs in locating
and utilizing relevant information from retrieved documents in RAG
applications.

扩展大型语言模型的上下文窗口到 128k 个标记或更多，然而，语言模型仍然存在位置偏见和难以使用上下文的中间部分的问题，我们通过指导语言模型分配更多的关注于选定的上下文片段来研究 LLM 的相对位置感知性和缓解不均衡关注的可行性，我们对基于位置和索引的指令进行了多文档问答任务的全面研究，发现语言模型对上下文的相对位置没有意识，但它们展示了使用匹配索引调整关注于特定片段的能力，我们的分析深化了对 LLM 中位置偏见的理解，并提供了通过指导缓解此偏见的途径，从而使 LLM 在 RAG 应用中定位和利用检索文档中的相关信息。