The learning and deployment of long-LLMs remains a challenging problem
despite recent progresses. In this work, we argue that the long-LLMs are not a
necessity to solve long-context tasks, as common long-context tasks are
short-context solvable, i.e. they can be solved by purely working with oracle
short-contexts within the long-context tasks' inputs. On top of this argument,
we propose a framework called LC-Boost (Long-Context Bootstrapper), which
enables a short-LLM to address the long-context tasks in a bootstrapping
manner. In our framework, the short-LLM prompts itself to reason for two
critical decisions: 1) how to access to the appropriate part of context within
the input, 2) how to make effective use of the accessed context. By adaptively
accessing and utilizing the context based on the presented tasks, LC-Boost can
serve as a general framework to handle diversified long-context processing
problems. We comprehensively evaluate different types of tasks from popular
long-context benchmarks, where LC-Boost is able to achieve a substantially
improved performance with a much smaller consumption of resource.

在这项工作中，我们认为虽然最近取得了一些进展，但长上下文语言模型（long-LLMs）的学习和部署仍然是一个具有挑战性的问题。我们提出了一个名为 LC-Boost（长上下文引导器）的框架，该框架使得短上下文语言模型（short-LLM）能够以自主引导的方式解决长上下文任务。通过根据给出的任务自适应地访问和利用上下文，LC-Boost 可以作为处理不同类型的长上下文处理问题的通用框架。我们从流行的长上下文基准中全面评估了不同类型的任务，在较小的资源消耗下，LC-Boost 能够实现显著提高的性能。

长上下文任务是否需要长 LLMs？

Are Long-LLMs A Necessity For Long-Context Tasks?

While many contemporary large language models (LLMs) can process lengthy
input, they still struggle to fully utilize information within the long
context, known as the lost-in-the-middle challenge. We hypothesize that it
stems from insufficient explicit supervision during the long-context training,
which fails to emphasize that any position in a long context can hold crucial
information. Based on this intuition, our study presents information-intensive
(IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer
dataset, where the answer requires (1) fine-grained information awareness on a
short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
and (2) the integration and reasoning of information from two or more short
segments. Through applying this information-intensive training on Mistral-7B,
we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
FILM-7B for utilizing long contexts, we design three probing tasks that
encompass various context styles (document, code, and structured-data context)
and information retrieval patterns (forward, backward, and bi-directional
retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
information from different positions in its 32K context window. Beyond these
probing tasks, FILM-7B significantly improves the performance on real-world
long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
accuracy on MMLU). Github Link: this https URL

通过信息密集型（IN2）训练，我们提出了一种完全数据驱动的解决方案来克服长上下文的缺失问题，该方法能够有效地利用长上下文中的信息，并且在现实世界的长上下文任务中显著提高性能。