While large language models (LLMs) are equipped with longer text input
capabilities than before, they are struggling to seek correct information in
long contexts. The "lost in the middle" problem challenges most LLMs, referring
to the dramatic decline in accuracy when correct information is located in the
middle. To overcome this crucial issue, this paper proposes to enhance the
information searching and reflection ability of LLMs in long contexts via
specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA).
Following these tasks, our model excels in focusing more precisely on the
desired information. Experimental results show substantial improvement in
Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7%
absolute gain in shuffled settings, by 21.5% in passage retrieval task. We
release our model, Ziya-Reader to promote related research in the community.

通过 Attention Strengthening Multi-doc QA 任务，本文提出了一种增强长文本上大语言模型（LLMs）信息搜索以及反思能力的方法，实验证明在 Multi-doc QA 和其他基准测试中相比最先进模型，我们的模型在乱序设置下取得了 13.7％绝对增益，以及在段落检索任务中取得了 21.5％的改进。我们将我们的模型 Ziya-Reader 发布出来，以促进相关研究在社区中的开展。