Long-context processing is a critical ability that constrains the
applicability of large language models. Although there exist various methods
devoted to enhancing the long-context processing ability of large language
models (LLMs), they are developed in an isolated manner and lack systematic
analysis and integration of their strengths, hindering further developments. In
this paper, we introduce UniMem, a unified framework that reformulates existing
long-context methods from the view of memory augmentation of LLMs. UniMem is
characterized by four key dimensions: Memory Management, Memory Writing, Memory
Reading, and Memory Injection, providing a systematic theory for understanding
various long-context methods. We reformulate 16 existing methods based on
UniMem and analyze four representative methods: Transformer-XL, Memorizing
Transformer, RMT, and Longformer into equivalent UniMem forms to reveal their
design principles and strengths. Based on these analyses, we propose UniMix, an
innovative approach that integrates the strengths of these algorithms.
Experimental results show that UniMix achieves superior performance in handling
long contexts with significantly lower perplexity than baselines.

提出了 UniMem 框架，将现有的长上下文处理方法从记忆增强的角度进行统一；通过分析 Transformer-XL、Memorizing Transformer、RMT 和 Longformer 等方法，揭示它们的设计原理和优势；基于这些分析，提出了整合这些算法优势的创新方法 UniMix，并通过实验证明了 UniMix 在处理长上下文方面具有比基准模型更低的困惑度。