In this paper, we introduce Positional Skip-wisE (PoSE) training for
efficient adaptation of large language models~(LLMs) to extremely long context
windows. PoSE decouples train length from target context window size by
simulating long inputs using a fixed context window with manipulated position
indices during training. Concretely, we select several short chunks from a long
input sequence, and introduce distinct skipping bias terms to modify the
position indices of each chunk. These bias terms, along with the length of each
chunk, are altered for each training example, allowing the model to adapt to
all positions within the target context window without training on full length
inputs. Experiments show that, compared with fine-tuning on the full length,
PoSE greatly reduces memory and time overhead with minimal impact on
performance. Leveraging this advantage, we have successfully extended the LLaMA
model to 128k tokens. Furthermore, we empirically confirm that PoSE is
compatible with all RoPE-based LLMs and various position interpolation
strategies. Notably, by decoupling fine-tuning length from target context
window, PoSE can theoretically extend the context window infinitely,
constrained only by memory usage for inference. With ongoing advancements for
efficient inference, we believe PoSE holds great promise for scaling the
context window even further.

本文介绍了 Positional Skip-wisE（PoSE）训练方法，用于将大型语言模型（LLMs）高效地适应于极长的上下文窗口。通过在训练过程中使用具有操作位置索引的固定上下文窗口来模拟长输入，PoSE 将训练长度与目标上下文窗口大小分离。实验证明，与在完整长度上进行微调相比，PoSE 大大降低了内存和时间开销，并对性能影响很小。利用这一优势，我们成功将 LLaMA 模型扩展到了 128k 标记。此外，我们在实验中证实了 PoSE 与所有基于 RoPE 的 LLMs 和各种位置插值策略是兼容的。值得注意的是，通过解耦微调长度和目标上下文窗口，PoSE 在理论上可以无限扩展上下文窗口，仅受推理内存使用的限制。随着高效推理技术的不断发展，我们相信 PoSE 在进一步扩展上下文窗口方面具有巨大的潜力。