In recent years, there have been remarkable advancements in the performance
of Transformer-based Large Language Models (LLMs) across various domains. As
these LLMs are deployed for increasingly complex tasks, they often face the
needs to conduct longer reasoning processes or understanding larger contexts.
In these situations, the length generalization failure of LLMs on long
sequences become more prominent. Most pre-training schemes truncate training
sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
generate fluent texts, let alone carry out downstream tasks, after longer
contexts, even with relative positional encoding which is designed to cope with
this problem. Common solutions such as finetuning on longer corpora often
involves daunting hardware and time costs and requires careful training process
design. To more efficiently leverage the generation capacity of existing LLMs,
we theoretically and empirically investigate the main out-of-distribution (OOD)
factors contributing to this problem. Inspired by this diagnosis, we propose a
simple yet effective solution for on-the-fly length generalization,
LM-Infinite, which involves only a $\Lambda$-shaped attention mask and a
distance limit while requiring no parameter updates or learning. We find it
applicable to a variety of LLMs using relative-position encoding methods.
LM-Infinite is computational efficient with $O(n)$ time and space, and
demonstrates consistent fluency and generation quality to as long as 32k tokens
on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
task such as passkey retrieval, it continues to work on inputs much longer than
training lengths where vanilla models fail immediately.

基于 Transformer 的大型语言模型存在长度泛化问题，传统的预训练模式需要耗费大量时间和资源进行解决，本研究提出了一种简单而有效的解决方案 LM-Infinite，不需要参数更新或学习，具有计算效率高且生成质量好的特点，在长序列上有较之前的模型更好的表现。