To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer

使用递归上下文压缩方法（RCC）有效地扩展了基于Transformer的大型语言模型(LLMs)的上下文窗口长度，解决了模型回应质量变差的问题，并以接近0.95的BLEU4得分在文本重建任务中实现了高达32倍的压缩率，以及在1M序列长度的密码检索任务中近乎100％的准确率，同时在长文本问答任务中表现出与非压缩方法相媲美的性能，并显著节省了存储资源。

循环上下文压缩：高效扩大LLM的上下文窗口