Mamba is an effective state space model with linear computation complexity.
It has recently shown impressive efficiency in dealing with high-resolution
inputs across various vision tasks. In this paper, we reveal that the powerful
Mamba model shares surprising similarities with linear attention Transformer,
which typically underperform conventional Transformer in practice. By exploring
the similarities and disparities between the effective Mamba and subpar linear
attention Transformer, we provide comprehensive analyses to demystify the key
factors behind Mamba's success. Specifically, we reformulate the selective
state space model and linear attention within a unified formulation, rephrasing
Mamba as a variant of linear attention Transformer with six major distinctions:
input gate, forget gate, shortcut, no attention normalization, single-head, and
modified block design. For each design, we meticulously analyze its pros and
cons, and empirically evaluate its impact on model performance in vision tasks.
Interestingly, the results highlight the forget gate and block design as the
core contributors to Mamba's success, while the other four designs are less
crucial. Based on these findings, we propose a Mamba-Like Linear Attention
(MLLA) model by incorporating the merits of these two key designs into linear
attention. The resulting model outperforms various vision Mamba models in both
image classification and high-resolution dense prediction tasks, while enjoying
parallelizable computation and fast inference speed. Code is available at
this https URL

探索了 Mamba 模型与线性注意力 Transformer 之间的相似性和差异性，发现忘记门和块设计是 Mamba 模型成功的核心因素，并通过融合这两个关键设计提出了 Mamba-Like Linear Attention (MLLA) 模型，该模型在图像分类和高分辨率密集预测任务中优于多种视觉任务中的 Mamba 模型，同时具有可并行计算和快速推理速度。