Extending large language models (LLMs) to process longer inputs is crucial
for numerous applications. However, the considerable computational cost of
transformers, coupled with limited generalization of positional encoding,
restricts the size of their context window. We introduce Context Expansion with
Parallel Encoding (CEPE), a framework that can be applied to any existing
decoder-only LLMs to extend their context window. CEPE adopts a small encoder
to process long inputs chunk by chunk and enables the frozen decoder to
leverage additional contexts via cross-attention. CEPE is efficient,
generalizable, and versatile: trained with 8K-token documents, CEPE extends the
context window of LLAMA-2 to 128K tokens, offering 10x the throughput with only
1/6 of the memory. CEPE yields strong performance on language modeling and
in-context learning. CEPE also excels in retrieval-augmented applications,
while existing long-context models degenerate with retrieved contexts. We
further introduce a CEPE variant that can extend the context window of
instruction-tuned models with only unlabeled data, and showcase its
effectiveness on LLAMA-2-CHAT, leading to a strong instruction-following model
that can leverage very long context on downstream tasks.

通过引入上下文扩展并行编码（CEPE）框架，可以将现有的仅解码的大型语言模型（LLMs）的上下文窗口扩展，使其能够更有效地处理长输入并且在检索增强应用中表现优异。