While large language models (LLMs) are equipped with longer text input
capabilities than before, they are struggling to seek correct information in
long contexts. The "lost in the middle" problem challenges most LLMs, referring
to the dramatic decline in accuracy when correct information is located in the
middle. To overcome this crucial issue, this paper proposes to enhance the
information searching and reflection ability of LLMs in long contexts via
specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA).
Following these tasks, our model excels in focusing more precisely on the
desired information. Experimental results show substantial improvement in
Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7%
absolute gain in shuffled settings, by 21.5% in passage retrieval task. We
release our model, Ziya-Reader to promote related research in the community.

通过 Attention Strengthening Multi-doc QA 任务，本文提出了一种增强长文本上大语言模型（LLMs）信息搜索以及反思能力的方法，实验证明在 Multi-doc QA 和其他基准测试中相比最先进模型，我们的模型在乱序设置下取得了 13.7％绝对增益，以及在段落检索任务中取得了 21.5％的改进。我们将我们的模型 Ziya-Reader 发布出来，以促进相关研究在社区中的开展。

中间不再迷失：通过增强注意力来提升大型语言模型在问答任务中的表现

Never Lost in the Middle: Improving Large Language Models via Attention  Strengthening Question Answering

Using natural language, Conversational Bot offers unprecedented ways to many
challenges in areas such as information searching, item recommendation, and
question answering. Existing bots are usually developed through retrieval-based
or generative-based approaches, yet both of them have their own advantages and
disadvantages. To assemble this two approaches, we propose a hybrid
retrieval-generation network (HeroNet) with the three-fold ideas: 1). To
produce high-quality sentence representations, HeroNet performs multi-task
learning on two subtasks: Similar Queries Discovery and Query-Response
Matching. Specifically, the retrieval performance is improved while the model
size is reduced by training two lightweight, task-specific adapter modules that
share only one underlying T5-Encoder model. 2). By introducing adversarial
training, HeroNet is able to solve both retrieval\&generation tasks
simultaneously while maximizing performance of each other. 3). The retrieval
results are used as prior knowledge to improve the generation performance while
the generative result are scored by the discriminator and their scores are
integrated into the generator's cross-entropy loss function. The experimental
results on a open dataset demonstrate the effectiveness of the HeroNet and our
code is available at this https URL

通过引入多任务学习和对抗训练，我们提出了一种混合检索 - 生成网络 (HeroNet)，该网络能够同时解决检索和生成任务，提高检索及生成的性能，有效地解决自然语言交互等领域中的信息搜索问题。