We present the Linear Complexity Sequence Model (LCSM), a comprehensive solution that unites various sequence modeling techniques with linear complexity, including linear attention, state space model, long convolution, and linear RNN, within a single framework. The goal is to enhance comprehension of these models by analyzing the impact of each component from a cohesive and streamlined viewpoint. Specifically, we segment the modeling processes of these models into three distinct stages: Expand, Oscillation, and Shrink (EOS), with each model having its own specific settings. The Expand stage involves projecting the input signal onto a high-dimensional memory state. This is followed by recursive operations performed on the memory state in the Oscillation stage. Finally, the memory state is projected back to a low-dimensional space in the Shrink stage. We perform comprehensive experiments to analyze the impact of different stage settings on language modeling and retrieval tasks. Our results show that data-driven methods are crucial for the effectiveness of the three stages in language modeling, whereas hand-crafted methods yield better performance in retrieval tasks.

我们提出了线性复杂度序列模型（LCSM），将各种序列建模技术（包括线性注意力、状态空间模型、长卷积和线性RNN等）以及展开、振荡和收缩（EOS）三个不同阶段的建模过程融入一个统一的框架中。通过从一个连贯而简化的视角分析每个组成部分的影响，我们旨在提高对这些模型的理解能力。通过广泛的实验证明，数据驱动方法对于语言建模的三个阶段的有效性至关重要，而手工制作的方法在检索任务中表现更佳。

从统一视角解开线性复杂度序列模型的秘密