Foundation models, now powering most of the exciting applications in deep
learning, are almost universally based on the Transformer architecture and its
core attention module. Many subquadratic-time architectures such as linear
attention, gated convolution and recurrent models, and structured state space
models (SSMs) have been developed to address Transformers' computational
inefficiency on long sequences, but they have not performed as well as
attention on important modalities such as language. We identify that a key
weakness of such models is their inability to perform content-based reasoning,
and make several improvements. First, simply letting the SSM parameters be
functions of the input addresses their weakness with discrete modalities,
allowing the model to selectively propagate or forget information along the
sequence length dimension depending on the current token. Second, even though
this change prevents the use of efficient convolutions, we design a
hardware-aware parallel algorithm in recurrent mode. We integrate these
selective SSMs into a simplified end-to-end neural network architecture without
attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5$\times$
higher throughput than Transformers) and linear scaling in sequence length, and
its performance improves on real data up to million-length sequences. As a
general sequence model backbone, Mamba achieves state-of-the-art performance
across several modalities such as language, audio, and genomics. On language
modeling, our Mamba-3B model outperforms Transformers of the same size and
matches Transformers twice its size, both in pretraining and downstream
evaluation.

基于 Transformer 架构的基础模型凭借其核心注意力模块，驱动着深度学习中大部分令人兴奋的应用。我们发现这种模型的一个关键弱点是其无法进行内容导向的推理，并对此进行了改进，通过让结构状态空间模型（SSMs）参数成为输入的函数来解决离散模态的弱点，该模型在长度可选的维度上选择性地传播或遗忘信息，并且通过在递归模式下设计一种硬件感知并行算法，将这些选择性 SSMs 集成到简化的端到端神经网络架构中。该模型（Mamba）具有快速推断速度（比 Transformers 快 5 倍）和序列长度的线性扩展，并在实际数据上对长达百万长度的序列显示出改进。作为一种基于通用序列模型的支持，Mamba 在语言、音频和基因组等多个模态上实现了最先进的性能。在语言建模中，我们的 Mamba-3B 模型在预训练和下游评估中均优于同样大小的 Transformers，与其两倍大小的模型性能相当。