Large language models (LLMs) call for extension of context to handle many
critical applications. However, the existing approaches are prone to expensive
costs and inferior quality of context extension. In this work, we propose
Extensible Embedding, which realizes high-quality extension of LLM's context
with strong flexibility and cost-effectiveness. Extensible embedding stand as
an enhancement of typical token embedding, which represents the information for
an extensible scope of context instead of a single token. By leveraging such
compact input units of higher information density, the LLM can access to a vast
scope of context even with a small context window. Extensible embedding is
systematically optimized in architecture and training method, which leads to
multiple advantages. 1) High flexibility of context extension, which flexibly
supports ad-hoc extension of diverse context lengths. 2) Strong sample
efficiency of training, which enables the embedding model to be learned in a
cost-effective way. 3) Superior compatibility with the existing LLMs, where the
extensible embedding can be seamlessly introduced as a plug-in component.
Comprehensive evaluations on long-context language modeling and understanding
tasks verify extensible embedding as an effective, efficient, flexible, and
compatible method to extend the LLM's context.

通过利用高信息密度的紧凑输入单元，可实现对大范围上下文的访问，使得可拓展嵌入成为高效、灵活且兼容的方法来扩展大语言模型的上下文。

可扩展的嵌入：适用于 LLM 上下文长度的灵活乘数

Extensible Embedding: A Flexible Multipler For LLM's Context Length

Large language models (LLMs) call for extension of context to handle many
critical applications. However, the existing approaches are prone to expensive
costs and inferior quality of context extension. In this work, we
proposeExtensible Embedding, which realizes high-quality extension of LLM's
context with strong flexibility and cost-effectiveness. Extensible embedding
stand as an enhancement of typical token embedding, which represents the
information for an extensible scope of context instead of a single token. By
leveraging such compact input units of higher information density, the LLM can
access to a vast scope of context even with a small context window. Extensible
embedding is systematically optimized in architecture and training method,
which leads to multiple advantages. 1) High flexibility of context extension,
which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong
sample efficiency of training, which enables the embedding model to be learned
in a cost-effective way. 3) Superior compatibility with the existing LLMs,
where the extensible embedding can be seamlessly introduced as a plug-in
component. Comprehensive evaluations on long-context language modeling and
understanding tasks verify extensible embedding as an effective, efficient,
flexible, and compatible method to extend the LLM's context.

通过可扩展嵌入实现了高质量、灵活、具有成本效益的扩展大型语言模型的上下文，通过优化架构和训练方法，具有上下文扩展的高灵活性、低成本的训练和与现有大型语言模型的兼容性，进而在长上下文语言建模和理解任务上进行综合评估，验证了可扩展嵌入作为一种有效、高效、灵活和兼容的扩展大型语言模型上下文的方法。

BGE 地标嵌入：用于检索增强的长上下文大语言模型的无分块嵌入方法

BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval  Augmented Long-Context Large Language Models

We present LongLoRA, an efficient fine-tuning approach that extends the
context sizes of pre-trained large language models (LLMs), with limited
computation cost. Typically, training LLMs with long context sizes is
computationally expensive, requiring extensive training hours and GPU
resources. For example, training on the context length of 8192 needs 16x
computational costs in self-attention layers as that of 2048. In this paper, we
speed up the context extension of LLMs in two aspects. On the one hand,
although dense global attention is needed during inference, fine-tuning the
model can be effectively and efficiently done by sparse local attention. The
proposed shift short attention effectively enables context extension, leading
to non-trivial computation saving with similar performance to fine-tuning with
vanilla attention. Particularly, it can be implemented with only two lines of
code in training, while being optional in inference. On the other hand, we
revisit the parameter-efficient fine-tuning regime for context expansion.
Notably, we find that LoRA for context extension works well under the premise
of trainable embedding and normalization. LongLoRA demonstrates strong
empirical results on various tasks on LLaMA2 models from 7B/13B to 70B.
LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a
single 8x A100 machine. LongLoRA extends models' context while retaining their
original architectures, and is compatible with most existing techniques, like
FlashAttention-2. In addition, to make LongLoRA practical, we collect a
dataset, LongQA, for supervised fine-tuning. It contains more than 3k long
context question-answer pairs.

LongLoRA 是一种高效的微调方法，通过有限的计算成本扩展了预训练大型语言模型（LLMs）的上下文大小。