Softmax attention is the principle backbone of foundation models for various
artificial intelligence applications, yet its quadratic complexity in sequence
length can limit its inference throughput in long-context settings. To address
this challenge, alternative architectures such as linear attention, State Space
Models (SSMs), and Recurrent Neural Networks (RNNs) have been considered as
more efficient alternatives. While connections between these approaches exist,
such models are commonly developed in isolation and there is a lack of
theoretical understanding of the shared principles underpinning these
architectures and their subtle differences, greatly influencing performance and
scalability. In this paper, we introduce the Dynamical Systems Framework (DSF),
which allows a principled investigation of all these architectures in a common
representation. Our framework facilitates rigorous comparisons, providing new
insights on the distinctive characteristics of each model class. For instance,
we compare linear attention and selective SSMs, detailing their differences and
conditions under which both are equivalent. We also provide principled
comparisons between softmax attention and other model classes, discussing the
theoretical conditions under which softmax attention can be approximated.
Additionally, we substantiate these new insights with empirical validations and
mathematical arguments. This shows the DSF's potential to guide the systematic
development of future more efficient and scalable foundation models.

基于动力系统框架的行为比较研究中，研究了基于 softmax attention、linear attention、State Space Models (SSMs) 和 Recurrent Neural Networks (RNNs) 的模型类之间的一些关键特性和条件以及它们在效率和可扩展性方面的差异。

理解基础模型的区别：注意力、状态空间模型和循环神经网络

Understanding the differences in Foundation Models: Attention, State  Space Models, and Recurrent Neural Networks

Point cloud videos effectively capture real-world spatial geometries and
temporal dynamics, which are essential for enabling intelligent agents to
understand the dynamically changing 3D world we live in. Although static 3D
point cloud processing has witnessed significant advancements, designing an
effective 4D point cloud video backbone remains challenging, mainly due to the
irregular and unordered distribution of points and temporal inconsistencies
across frames. Moreover, recent state-of-the-art 4D backbones predominantly
rely on transformer-based architectures, which commonly suffer from large
computational costs due to their quadratic complexity, particularly when
processing long video sequences. To address these challenges, we propose a
novel 4D point cloud video understanding backbone based on the recently
advanced State Space Models (SSMs). Specifically, our backbone begins by
disentangling space and time in raw 4D sequences, and then establishing
spatio-temporal correlations using our newly developed Intra-frame Spatial
Mamba and Inter-frame Temporal Mamba blocks. The Intra-frame Spatial Mamba
module is designed to encode locally similar or related geometric structures
within a certain temporal searching stride, which can effectively capture
short-term dynamics. Subsequently, these locally correlated tokens are
delivered to the Inter-frame Temporal Mamba module, which globally integrates
point features across the entire video with linear complexity, further
establishing long-range motion dependencies. Experimental results on human
action recognition and 4D semantic segmentation tasks demonstrate the
superiority of our proposed method. Especially, for long video sequences, our
proposed Mamba-based method has an 87.5% GPU memory reduction, 5.36 times
speed-up, and much higher accuracy (up to +10.4%) compared with
transformer-based counterparts on MSR-Action3D dataset.

基于最近提出的状态空间模型（SSMs）的新型 4D 点云视频理解骨干网络，通过解开原始 4D 序列中的空间和时间，使用新开发的 Intra-frame Spatial Mamba 和 Inter-frame Temporal Mamba 模块建立了点云视频内部和帧间的时空关联，并在人体动作识别和 4D 语义分割任务上证明了该方法的优越性。

MAMBA4D: 高效的长序列点云视频理解及解耦空间 - 时间状态空间模型

MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with  Disentangled Spatial-Temporal State Space Models

Transformers have widely adopted attention networks for sequence mixing and
MLPs for channel mixing, playing a pivotal role in achieving breakthroughs
across domains. However, recent literature highlights issues with attention
networks, including low inductive bias and quadratic complexity concerning
input sequence length. State Space Models (SSMs) like S4 and others (Hippo,
Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address
the above issues to help handle longer sequence lengths. Mamba, while being the
state-of-the-art SSM, has a stability issue when scaled to large networks for
computer vision datasets. We propose SiMBA, a new architecture that introduces
Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations
and uses the Mamba block for sequence modeling. Extensive performance studies
across image and time-series benchmarks demonstrate that SiMBA outperforms
existing SSMs, bridging the performance gap with state-of-the-art transformers.
Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet
and transfer learning benchmarks such as Stanford Car and Flower as well as
task learning benchmarks as well as seven time series benchmark datasets. The
project page is available on this website
~https://github.com/badripatro/Simba.

提出了一种新的体系结构 SiMBA，通过特定特征值计算引入 Einstein FFT（EinFFT）用于通道建模，使用 Mamba 块进行序列建模，通过对图像和时间序列基准的广泛性能研究证明 SiMBA 在性能上优于现有的 SSMs，缩小与现有的 transformers 的性能差距。

SiMBA: 简化的基于 Mamba 的视觉与多元时间序列架构

SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate  Time series

In the realm of medical image segmentation, both CNN-based and
Transformer-based models have been extensively explored. However, CNNs exhibit
limitations in long-range modeling capabilities, whereas Transformers are
hampered by their quadratic computational complexity. Recently, State Space
Models (SSMs), exemplified by Mamba, have emerged as a promising approach. They
not only excel in modeling long-range interactions but also maintain a linear
computational complexity. In this paper, leveraging state space models, we
propose a U-shape architecture model for medical image segmentation, named
Vision Mamba UNet (VM-UNet). Specifically, the Visual State Space (VSS) block
is introduced as the foundation block to capture extensive contextual
information, and an asymmetrical encoder-decoder structure is constructed. We
conduct comprehensive experiments on the ISIC17, ISIC18, and Synapse datasets,
and the results indicate that VM-UNet performs competitively in medical image
segmentation tasks. To our best knowledge, this is the first medical image
segmentation model constructed based on the pure SSM-based model. We aim to
establish a baseline and provide valuable insights for the future development
of more efficient and effective SSM-based segmentation systems. Our code is
available at this https URL

本研究提出了一种基于状态空间模型的医学图像分割模型 Vision Mamba UNet (VM-UNet)，通过引入 Visual State Space (VSS) 块以捕获广泛的上下文信息，并构建了一个不对称的编码器 - 解码器结构。实验结果表明，VM-UNet 在医学图像分割任务中具有竞争力。此外，该模型是首个基于纯 SSM 模型构建的医学图像分割模型，旨在为未来开发更高效、更有效的 SSM 分割系统奠定基础并提供有价值的见解。