Sequence modeling is a crucial area across various domains, including Natural
Language Processing (NLP), speech recognition, time series forecasting, music
generation, and bioinformatics. Recurrent Neural Networks (RNNs) and Long Short
Term Memory Networks (LSTMs) have historically dominated sequence modeling
tasks like Machine Translation, Named Entity Recognition (NER), etc. However,
the advancement of transformers has led to a shift in this paradigm, given
their superior performance. Yet, transformers suffer from $O(N^2)$ attention
complexity and challenges in handling inductive bias. Several variations have
been proposed to address these issues which use spectral networks or
convolutions and have performed well on a range of tasks. However, they still
have difficulty in dealing with long sequences. State Space Models(SSMs) have
emerged as promising alternatives for sequence modeling paradigms in this
context, especially with the advent of S4 and its variants, such as S4nd,
Hippo, Hyena, Diagnol State Spaces (DSS), Gated State Spaces (GSS), Linear
Recurrent Unit (LRU), Liquid-S4, Mamba, etc. In this survey, we categorize the
foundational SSMs based on three paradigms namely, Gating architectures,
Structural architectures, and Recurrent architectures. This survey also
highlights diverse applications of SSMs across domains such as vision, video,
audio, speech, language (especially long sequence modeling), medical (including
genomics), chemical (like drug design), recommendation systems, and time series
analysis, including tabular data. Moreover, we consolidate the performance of
SSMs on benchmark datasets like Long Range Arena (LRA), WikiText, Glue, Pile,
ImageNet, Kinetics-400, sstv2, as well as video datasets such as Breakfast,
COIN, LVU, and various time series datasets. The project page for Mamba-360
work is available on this webpage.https://github.com/badripatro/mamba360.

序列建模是跨多个领域的一个关键领域，包括自然语言处理、语音识别、时间序列预测、音乐生成和生物信息学。历史上，循环神经网络（RNNs）和长短时记忆网络（LSTMs）在机器翻译、命名实体识别等序列建模任务中占主导地位，但转换器的进步改变了这一范式，因为其性能更优越。然而，转换器面临 O (N^2) 注意力复杂度和处理归纳偏差的挑战。各种改进方法已被提出来应对这些问题，其中使用频谱网络或卷积在一些任务上表现良好，但仍然难以处理长序列。状态空间模型（SSMs）已经成为此背景下序列建模范式的有希望的可替代选择，特别是随着 S4 及其变种（如 S4nd、Hippo、Hyena、Diagnol State Spaces（DSS）、Gated State Spaces（GSS）、Linear Recurrent Unit（LRU）、Liquid-S4、Mamba 等）的出现。本综述将基于门控结构、结构体系和循环结构对基础 SSMs 进行分类，还重点介绍了 SSMs 在视觉、视频、音频、语音、语言（特别是长序列建模）、医疗（包括基因组学）、化学（如药物设计）、推荐系统和时间序列分析等领域的不同应用。此外，我们总结了 SSMs 在长序列竞技场（LRA）、WikiText、Glue、Pile、ImageNet、Kinetics-400、sstv2 以及早餐、硬币、LVU 等各种时间序列数据集上的性能。有关 Mamba-360 工作的项目页面可在此网页上找到：https://github.com/badripatro/mamba360。