Long-context modeling presents a significant challenge for transformer-based
large language models (LLMs) due to the quadratic complexity of the
self-attention mechanism and issues with length extrapolation caused by
pretraining exclusively on short inputs. Existing methods address computational
complexity through techniques such as text chunking, the kernel approach, and
structured attention, and tackle length extrapolation problems through
positional encoding, continued pretraining, and data engineering. These
approaches typically require $\textbf{sequential access}$ to the document,
necessitating reading from the first to the last token. We contend that for
goal-oriented reading of long documents, such sequential access is not
necessary, and a proficiently trained model can learn to omit hundreds of less
pertinent tokens. Inspired by human reading behaviors and existing empirical
observations, we propose $\textbf{random access}$, a novel reading strategy
that enables transformers to efficiently process long documents without
examining every token. Experimental results from pretraining, fine-tuning, and
inference phases validate the efficacy of our method.

长文本建模提出了对基于 Transformer 的大型语言模型（LLMs）的重大挑战，我们提出了一种新的阅读策略，即随机访问，可以使 Transformer 模型在处理长文档时高效地跳过不相关的标记。通过预训练、微调和推理阶段的实验证明了我们方法的有效性。