Sequence modeling is a crucial area across various domains, including Natural
Language Processing (NLP), speech recognition, time series forecasting, music
generation, and bioinformatics. Recurrent Neural Networks (RNNs) and Long Short
Term Memory Networks (LSTMs) have historically dominated sequence modeling
tasks like Machine Translation, Named Entity Recognition (NER), etc. However,
the advancement of transformers has led to a shift in this paradigm, given
their superior performance. Yet, transformers suffer from $O(N^2)$ attention
complexity and challenges in handling inductive bias. Several variations have
been proposed to address these issues which use spectral networks or
convolutions and have performed well on a range of tasks. However, they still
have difficulty in dealing with long sequences. State Space Models(SSMs) have
emerged as promising alternatives for sequence modeling paradigms in this
context, especially with the advent of S4 and its variants, such as S4nd,
Hippo, Hyena, Diagnol State Spaces (DSS), Gated State Spaces (GSS), Linear
Recurrent Unit (LRU), Liquid-S4, Mamba, etc. In this survey, we categorize the
foundational SSMs based on three paradigms namely, Gating architectures,
Structural architectures, and Recurrent architectures. This survey also
highlights diverse applications of SSMs across domains such as vision, video,
audio, speech, language (especially long sequence modeling), medical (including
genomics), chemical (like drug design), recommendation systems, and time series
analysis, including tabular data. Moreover, we consolidate the performance of
SSMs on benchmark datasets like Long Range Arena (LRA), WikiText, Glue, Pile,
ImageNet, Kinetics-400, sstv2, as well as video datasets such as Breakfast,
COIN, LVU, and various time series datasets. The project page for Mamba-360
work is available on this webpage.https://github.com/badripatro/mamba360.

序列建模是跨多个领域的一个关键领域，包括自然语言处理、语音识别、时间序列预测、音乐生成和生物信息学。历史上，循环神经网络（RNNs）和长短时记忆网络（LSTMs）在机器翻译、命名实体识别等序列建模任务中占主导地位，但转换器的进步改变了这一范式，因为其性能更优越。然而，转换器面临 O (N^2) 注意力复杂度和处理归纳偏差的挑战。各种改进方法已被提出来应对这些问题，其中使用频谱网络或卷积在一些任务上表现良好，但仍然难以处理长序列。状态空间模型（SSMs）已经成为此背景下序列建模范式的有希望的可替代选择，特别是随着 S4 及其变种（如 S4nd、Hippo、Hyena、Diagnol State Spaces（DSS）、Gated State Spaces（GSS）、Linear Recurrent Unit（LRU）、Liquid-S4、Mamba 等）的出现。本综述将基于门控结构、结构体系和循环结构对基础 SSMs 进行分类，还重点介绍了 SSMs 在视觉、视频、音频、语音、语言（特别是长序列建模）、医疗（包括基因组学）、化学（如药物设计）、推荐系统和时间序列分析等领域的不同应用。此外，我们总结了 SSMs 在长序列竞技场（LRA）、WikiText、Glue、Pile、ImageNet、Kinetics-400、sstv2 以及早餐、硬币、LVU 等各种时间序列数据集上的性能。有关 Mamba-360 工作的项目页面可在此网页上找到：https://github.com/badripatro/mamba360。

Mamba-360：状态空间模型作为长序列建模的 Transformer 替代方案调研：方法、应用和挑战

Mamba-360: Survey of State Space Models as Transformer Alternative for  Long Sequence Modelling: Methods, Applications, and Challenges

Sparse Transformer has recently attracted a lot of attention since the
ability for reducing the quadratic dependency on the sequence length. We argue
that two factors, information bottleneck sensitivity and inconsistency between
different attention topologies, could affect the performance of the Sparse
Transformer. This paper proposes a well-designed model named ERNIE-Sparse. It
consists of two distinctive parts: (i) Hierarchical Sparse Transformer (HST) to
sequentially unify local and global information. (ii) Self-Attention
Regularization (SAR) method, a novel regularization designed to minimize the
distance for transformers with different attention topologies. To evaluate the
effectiveness of ERNIE-Sparse, we perform extensive evaluations. Firstly, we
perform experiments on a multi-modal long sequence modeling task benchmark,
Long Range Arena (LRA). Experimental results demonstrate that ERNIE-Sparse
significantly outperforms a variety of strong baseline methods including the
dense attention and other efficient sparse attention methods and achieves
improvements by 2.77% (57.78% vs. 55.01%). Secondly, to further show the
effectiveness of our method, we pretrain ERNIE-Sparse and verified it on 3 text
classification and 2 QA downstream tasks, achieve improvements on
classification benchmark by 0.83% (92.46% vs. 91.63%), on QA benchmark by 3.24%
(74.67% vs. 71.43%). Experimental results continue to demonstrate its superior
performance.

提出 ERNIE-Sparse 模型，它包含两个不同的部分：分层稀疏 Transformer（Hierarchical Sparse Transformer，HST）和自注意力正则化（Self-Attention Regularization，SAR）方法。实验证明，在长序列建模任务和文本分类任务中，ERNIE-Sparse 显著优于其他基线方法。