This paper addresses the challenge of train-short-test-long (TSTL) scenarios
in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
where models pre-trained on shorter sequences face difficulty with
out-of-distribution (OOD) token positions in longer sequences. We introduce
Resonance RoPE, a novel approach designed to narrow the generalization gap in
TSTL scenarios by refining the interpolation of RoPE features for OOD
positions, significantly improving the model performance without additional
online computational costs. Furthermore, we present PosGen, a new synthetic
benchmark specifically designed for fine-grained behavior analysis in TSTL
scenarios, aiming to isolate the constantly increasing difficulty of token
generation on long contexts from the challenges of recognizing new token
positions. Our experiments on synthetic tasks show that after applying
Resonance RoPE, Transformers recognize OOD position better and more robustly.
Our extensive LLM experiments also show superior performance after applying
Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
both upstream language modeling tasks and a variety of downstream long-text
applications.

应对在大规模语言模型中的训练短测试长场景下，使用旋转位置嵌入（RoPE）时面临的挑战，本论文引入了共振 RoPE，一种旨在通过对 OOD 位置的 RoPE 特征插值进行精化，显著提高模型性能而无需额外在线计算成本的新方法。另外，我们提出了 PosGen，这是一个专门为 TSTL 场景中的细粒度行为分析设计的合成基准，旨在从生成长上下文的令牌的不断增加难度与识别新令牌位置的挑战中分离出来。我们在合成任务上的实验证明，在应用了共振 RoPE 之后，Transformer 在识别 OOD 位置方面表现更好、更稳健。我们的大量 LLM 实验也表明，在上游语言建模任务和各种下游长文本应用中，应用共振 RoPE 到当前最先进的 RoPE 缩放方法 YaRN 后，性能更优秀。