Structured state-space models (SSMs) such as S4, stemming from the seminal
work of Gu et al., are gaining popularity as effective approaches for modeling
sequential data. Deep SSMs demonstrate outstanding performance across a diverse
set of domains, at a reduced training and inference cost compared to
attention-based transformers. Recent developments show that if the linear
recurrence powering SSMs allows for multiplicative interactions between inputs
and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture
can surpass in both in accuracy and efficiency attention-powered foundation
models trained on text, at scales of billion parameters. In this paper, we give
theoretical grounding to this recent finding using tools from Rough Path
Theory: we show that when random linear recurrences are equipped with simple
input-controlled transitions (selectivity mechanism), then the hidden state is
provably a low-dimensional projection of a powerful mathematical object called
the signature of the input -- capturing non-linear interactions between tokens
at distinct timescales. Our theory not only motivates the success of modern
selective state-space models such as Mamba but also provides a solid framework
to understand the expressive power of future SSM variants.

基于 Rough Path Theory，本研究证明了具备选择性机制的随机线性递归在输入控制条件下可产生低维投射的隐藏状态，并阐述了现代选择性状态空间模型的成功和未来 SSM 变体的表达能力。

深度选择状态空间模型的理论基础

Theoretical Foundations of Deep Selective State-Space Models

The goal of this paper is to provide a system identification-friendly
introduction to the Structured State-space Models (SSMs). These models have
become recently popular in the machine learning community since, owing to their
parallelizability, they can be efficiently and scalably trained to tackle
extremely-long sequence classification and regression problems. Interestingly,
SSMs appear as an effective way to learn deep Wiener models, which allows to
reframe SSMs as an extension of a model class commonly used in system
identification. In order to stimulate a fruitful exchange of ideas between the
machine learning and system identification communities, we deem it useful to
summarize the recent contributions on the topic in a structured and accessible
form. At last, we highlight future research directions for which this community
could provide impactful contributions.

为了促进机器学习和系统识别社区之间的有效交流，本文旨在以系统辨识友好的方式介绍结构化状态空间模型（SSMs）。SSMs 近期在机器学习社区受到欢迎，因为它们可以高效可扩展地训练以解决极长序列分类和回归问题。此外，SSMs 还被视为学习深威纳模型的有效方法，从而将 SSMs 作为一种常用于系统辨识的模型类进行扩展。最后，本文总结了该主题的最新贡献并强调了未来的研究方向。