Typically, training LLMs with long context sizes is computationally
expensive, requiring extensive training hours and GPU resources. Existing
long-context extension methods usually need additional training procedures to
support corresponding long-context windows, where the long-context training
data (e.g., 32k) is needed, and high GPU training costs are assumed. To address
the aforementioned issues, we propose an Efficient and Extreme length extension
method for Large Language Models, called E 2 -LLM, with only one training
procedure and dramatically reduced computation cost, which also removes the
need to collect long-context data. Concretely, first, the training data of our
E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost
greatly. Second, the training procedure on the short training context window is
performed only once time, and we can support different evaluation context
windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings,
we introduce two different augmentation methods on the scale and position index
parameters for different samples in training. It aims to make the model more
robust to the different relative differences when directly interpolating the
arbitrary context length at inference. Comprehensive experimental results on
multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on
challenging long-context tasks.

我们提出了一种名为 E2-LLM 的高效和极长扩展的大型语言模型方法，通过减少计算成本并对不同样本进行增强方法来在推理时支持任意上下文长度，实验结果表明其在具有挑战性的长上下文任务中的有效性。