Mamba state-space models (SSMs) have recently outperformed state-of-the-art
(SOTA) Transformer large language models (LLMs) in various tasks and been
widely adapted. However, Mamba's downstream learning capabilities remain either
unexplored$\unicode{x2013}$e.g., mixed-precision (MPFT) and parameter-efficient
fine-tuning (PEFT)--or under-evaluated$\unicode{x2013}$e.g., in-context
learning (ICL). For the latter, recent works reported Mamba's ICL rivals SOTA
Transformer LLMs using non-standard benchmarks. In contrast, we show that on
standard benchmarks, pretrained Mamba models achieve only 38% of the ICL
performance improvements (over zero-shot) of comparable Transformers.
Enabling MPFT and PEFT in Mamba architectures is challenging due to recurrent
dynamics and highly customized CUDA kernels, respectively. However, we prove
that Mamba's recurrent dynamics are robust to small input changes using
dynamical systems theory. Empirically, we show that performance changes in
Mamba's inference and fine-tuning due to mixed-precision align with Transformer
LLMs. Furthermore, we show that targeting key memory buffers in Mamba's
customized CUDA kernels for low-rank adaptation regularizes SSM parameters,
thus achieving parameter efficiency while retaining speedups. We show that
combining MPFT and PEFT enables up to 2.15 times more tokens-per-second and
65.5% reduced per-token-memory compared to full Mamba fine-tuning, while
achieving up to 81.5% of the ICL performance improvements (over zero-shot) of
comparably fine-tuned Transformers.

研究论文通过对 Mamba 状态空间模型进行多种实验和评估，证明了其在混合精度和参数高效微调方面的能力，同时还分析了其在上下文学习方面的性能与 Transformer 大型语言模型的差异，并提出了基于动力学系统理论的验证方法。