This work introduces an efficient method to scale Transformer-based Large
Language Models (LLMs) to infinitely long inputs with bounded memory and
computation. A key component in our proposed approach is a new attention
technique dubbed Infini-attention. The Infini-attention incorporates a
compressive memory into the vanilla attention mechanism and builds in both
masked local attention and long-term linear attention mechanisms in a single
Transformer block. We demonstrate the effectiveness of our approach on
long-context language modeling benchmarks, 1M sequence length passkey context
block retrieval and 500K length book summarization tasks with 1B and 8B LLMs.
Our approach introduces minimal bounded memory parameters and enables fast
streaming inference for LLMs.

该研究介绍了一种有效的方法，用于将基于 Transformer 的大型语言模型扩展到无限长的输入，同时保证有界的内存和计算。我们提出的方法的关键组成部分是一种称为 Infini-attention 的新的注意力技术，它将压缩性记忆融入到传统的注意力机制中，并在单个 Transformer 块中集成了被屏蔽的局部注意力和长期线性注意力机制。我们在长文本语言建模、1M 序列长度密钥上下文块检索和 500K 长度的书籍摘要任务上展示了我们方法的有效性，使用 1B 和 8B 规模的大型语言模型。我们的方法引入了最小化的有界内存参数，并实现了 LLMs 的快速流式推理。

高效的无限上下问 Transformer 与无限注意力

Leave No Context Behind: Efficient Infinite Context Transformers with  Infini-attention

Large language models (LLMs) are in need of sufficient contexts to handle
many critical applications, such as retrieval augmented generation and few-shot
learning. However, due to the constrained window size, the LLMs can only access
to the information within a limited context. Although the size of context
window can be extended by fine-tuning, it will result in a substantial cost in
both training and inference stage. In this paper, we present Extensible
Tokenization as an alternative method which realizes the flexible scaling of
LLMs' context. Extensible Tokenization stands as a midware in between of the
tokenized context and the LLM, which transforms the raw token embeddings into
the extensible embeddings. Such embeddings provide a more compact
representation for the long context, on top of which the LLM is able to
perceive more information with the same context window. Extensible Tokenization
is also featured by its flexibility: the scaling factor can be flexibly
determined within a feasible scope, leading to the extension of an arbitrary
context length at the inference time. Besides, Extensible Tokenization is
introduced as a drop-in component, which can be seamlessly plugged into not
only the LLM itself and but also its fine-tuned derivatives, bringing in the
extended contextual information while fully preserving the LLM's existing
capabilities. We perform comprehensive experiments on long-context language
modeling and understanding tasks, which verify Extensible Tokenization as an
effective, efficient, flexible, and compatible method to extend LLM's context.
Our model and source code will be made publicly available.

本文介绍了可拓展分词作为一种可选方法，可实现大语言模型（LLMs）上下文的灵活扩展，以提供更多信息。经过综合实验证明，可拓展分词是一种有效、高效、灵活和兼容的方法，可扩展 LLMs 的上下文。