While many contemporary large language models (LLMs) can process lengthy
input, they still struggle to fully utilize information within the long
context, known as the lost-in-the-middle challenge. We hypothesize that it
stems from insufficient explicit supervision during the long-context training,
which fails to emphasize that any position in a long context can hold crucial
information. Based on this intuition, our study presents information-intensive
(IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer
dataset, where the answer requires (1) fine-grained information awareness on a
short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
and (2) the integration and reasoning of information from two or more short
segments. Through applying this information-intensive training on Mistral-7B,
we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
FILM-7B for utilizing long contexts, we design three probing tasks that
encompass various context styles (document, code, and structured-data context)
and information retrieval patterns (forward, backward, and bi-directional
retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
information from different positions in its 32K context window. Beyond these
probing tasks, FILM-7B significantly improves the performance on real-world
long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
accuracy on MMLU). Github Link: this https URL

通过信息密集型（IN2）训练，我们提出了一种完全数据驱动的解决方案来克服长上下文的缺失问题，该方法能够有效地利用长上下文中的信息，并且在现实世界的长上下文任务中显著提高性能。

充分利用上下文的 LLM

Make Your LLM Fully Utilize the Context

This paper aims to overcome the "lost-in-the-middle" challenge of large
language models (LLMs). While recent advancements have successfully enabled
LLMs to perform stable language modeling with up to 4 million tokens, the
persistent difficulty faced by most LLMs in identifying relevant information
situated in the middle of the context has not been adequately tackled. To
address this problem, this paper introduces Multi-scale Positional Encoding
(Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the
capacity of LLMs to handle the relevant information located in the middle of
the context, without fine-tuning or introducing any additional overhead. Ms-PoE
leverages the position indice rescaling to relieve the long-term decay effect
introduced by RoPE, while meticulously assigning distinct scaling ratios to
different attention heads to preserve essential knowledge learned during the
pre-training step, forming a multi-scale context fusion from short to long
distance. Extensive experiments with a wide range of LLMs demonstrate the
efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of
up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are
available at this https URL

本文介绍了一种名为多尺度位置编码（Ms-PoE）的简单而有效的插件式方法，用于增强大型语言模型（LLMs）处理位于上下文中部的相关信息的能力，并通过广泛的实验证明了该方法的有效性。