Recently, with the emergence of numerous Large Language Models (LLMs), the
implementation of AI has entered a new era. Irrespective of these models' own
capacity and structure, there is a growing demand for LLMs to possess enhanced
comprehension of longer and more complex contexts with relatively smaller
sizes. Models often encounter an upper limit when processing sequences of
sentences that extend beyond their comprehension capacity and result in
off-topic or even chaotic responses. While several recent works attempt to
address this issue in various ways, they rarely focus on "why models are unable
to compensate or strengthen their capabilities on their own". In this paper, we
thoroughly investigate the nature of information transfer within LLMs and
propose a novel technique called Attention Transition. This technique empowers
models to achieve longer and better context comprehension with minimal
additional training or impact on generation fluency. Our experiments are
conducted in XSum and achieve substantial improvement compared with the
original generation results.

最近，随着大量的大型语言模型（LLMs）的出现，人工智能的实施进入了一个新时代。尽管这些模型本身具有能力和结构，但对于 LLMs 来说，能够对更长、更复杂的上下文具有增强理解能力而又保持相对较小的规模的需求不断增长。本文在对 LLMs 内部信息传递的本质进行深入研究的基础上，提出了一种名为 Attention Transition 的新技术。该技术使模型能够在最小的额外训练或对生成流畅性的影响下，实现更长、更好的上下文理解。我们的实验在 XSum 上进行，与原始生成结果相比取得了显著改进。