This paper addresses the challenge of train-short-test-long (TSTL) scenarios
in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
where models pre-trained on shorter sequences face difficulty with
out-of-distribution (OOD) token positions in longer sequences. We introduce
Resonance RoPE, a novel approach designed to narrow the generalization gap in
TSTL scenarios by refining the interpolation of RoPE features for OOD
positions, significantly improving the model performance without additional
online computational costs. Furthermore, we present PosGen, a new synthetic
benchmark specifically designed for fine-grained behavior analysis in TSTL
scenarios, aiming to isolate the constantly increasing difficulty of token
generation on long contexts from the challenges of recognizing new token
positions. Our experiments on synthetic tasks show that after applying
Resonance RoPE, Transformers recognize OOD position better and more robustly.
Our extensive LLM experiments also show superior performance after applying
Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
both upstream language modeling tasks and a variety of downstream long-text
applications.

应对在大规模语言模型中的训练短测试长场景下，使用旋转位置嵌入（RoPE）时面临的挑战，本论文引入了共振 RoPE，一种旨在通过对 OOD 位置的 RoPE 特征插值进行精化，显著提高模型性能而无需额外在线计算成本的新方法。另外，我们提出了 PosGen，这是一个专门为 TSTL 场景中的细粒度行为分析设计的合成基准，旨在从生成长上下文的令牌的不断增加难度与识别新令牌位置的挑战中分离出来。我们在合成任务上的实验证明，在应用了共振 RoPE 之后，Transformer 在识别 OOD 位置方面表现更好、更稳健。我们的大量 LLM 实验也表明，在上游语言建模任务和各种下游长文本应用中，应用共振 RoPE 到当前最先进的 RoPE 缩放方法 YaRN 后，性能更优秀。

共振 RoPE: 改善大型语言模型的上下文长度通用性

Resonance RoPE: Improving Context Length Generalization of Large  Language Models

The insurance industry is shifting their sales mode from offline to online,
in expectation to reach massive potential customers in the digitization era.
Due to the complexity and the nature of insurance products, a cost-effective
online sales solution is to exploit chatbot AI to raise customers' attention
and pass those with interests to human agents for further sales. For high
response and conversion rates of customers, it is crucial for the chatbot to
initiate a conversation with personalized opening sentences, which are
generated with user-specific topic selection and ordering. Such personalized
opening sentence generation is challenging because (i) there are limited
historical samples for conversation topic recommendation in online insurance
sales and (ii) existing text generation schemes often fail to support
customized topic ordering based on user preferences. We design POSGen, a
personalized opening sentence generation scheme dedicated for online insurance
sales. It transfers user embeddings learned from auxiliary online user
behaviours to enhance conversation topic recommendation, and exploits a context
management unit to arrange the recommended topics in user-specific ordering for
opening sentence generation. POSGen is deployed on a real-world online
insurance platform. It achieves 2.33x total insurance premium improvement
through a two-month global test.

该研究介绍了一种名为 POSGen 的个性化聊天机器人开场话生成方案，将用户嵌入式学习应用于保险销售话术，以改善传统线下保险销售转为在线保险销售后的客户互动体验，经过实际测试获得了 2.33 倍的总保费提高。