This paper aims to overcome the "lost-in-the-middle" challenge of large
language models (LLMs). While recent advancements have successfully enabled
LLMs to perform stable language modeling with up to 4 million tokens, the
persistent difficulty faced by most LLMs in identifying relevant information
situated in the middle of the context has not been adequately tackled. To
address this problem, this paper introduces Multi-scale Positional Encoding
(Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the
capacity of LLMs to handle the relevant information located in the middle of
the context, without fine-tuning or introducing any additional overhead. Ms-PoE
leverages the position indice rescaling to relieve the long-term decay effect
introduced by RoPE, while meticulously assigning distinct scaling ratios to
different attention heads to preserve essential knowledge learned during the
pre-training step, forming a multi-scale context fusion from short to long
distance. Extensive experiments with a wide range of LLMs demonstrate the
efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of
up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are
available at this https URL

本文介绍了一种名为多尺度位置编码（Ms-PoE）的简单而有效的插件式方法，用于增强大型语言模型（LLMs）处理位于上下文中部的相关信息的能力，并通过广泛的实验证明了该方法的有效性。