This paper presents a novel context compression method for Transformer
language models in online scenarios such as ChatGPT, where the context
continually expands. As the context lengthens, the attention process requires
more memory and computational resources, which in turn reduces the throughput
of the language model. To this end, we propose a compressed context memory
system that continually compresses the growing context into a compact memory
space. The compression process simply involves integrating a lightweight
conditional LoRA into the language model's forward pass during inference. Based
on the compressed context memory, the language model can perform inference with
reduced memory and attention operations. Through evaluations on conversation,
personalization, and multi-task learning, we demonstrate that our approach
achieves the performance level of a full context model with $5\times$ smaller
context memory space. Codes are available at
this https URL

本文提出一种在在线场景（如 ChatGPT）中用于 Transformer 语言模型的新型上下文压缩方法，将不断扩展的上下文压缩到紧凑的记忆空间中，并通过轻量级条件 LoRA 在推理过程中实现对压缩上下文记忆的操作，从而减小内存和注意力操作，实现与完整上下文模型相当的性能，但所需上下文内存空间仅为原来的五分之一。