Transformer-based Large Language Models (LLMs) are pioneering advances in
many natural language processing tasks, however, their exceptional capabilities
are restricted within the preset context window of Transformer. Position
Embedding (PE) scaling methods, while effective in extending the context window
to a specific length, demonstrate either notable limitations in their
extrapolation abilities or sacrificing partial performance within the context
window. Length extrapolation methods, although theoretically capable of
extending the context window beyond the training sequence length, often
underperform in practical long-context applications. To address these
challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We
generalise the PE scaling approaches to model the continuous dynamics by
ordinary differential equations over the length scaling factor, thereby
overcoming the constraints of current PE scaling methods designed for specific
lengths. Moreover, by extending the dynamics to desired context lengths beyond
the training sequence length, CLEX facilitates the length extrapolation with
impressive performance in practical tasks. We demonstrate that CLEX can be
seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such
as LLaMA and GPT-NeoX, with negligible impact on training and inference
latency. Experimental results reveal that CLEX can effectively extend the
context window to over 4x or almost 8x training length, with no deterioration
in performance. Furthermore, when evaluated on the practical LongBench
benchmark, our model trained on a 4k length exhibits competitive performance
against state-of-the-art open-source models trained on context lengths up to
32k.

建议一种基于连续长度外推（CLEX）的 Transformer-based Large Language Models (LLMs)，可将 context window 扩展到训练序列长度的 4 倍或 8 倍，并在实际任务中表现出竞争性性能。