Structured state-space models (SSMs) such as S4, stemming from the seminal
work of Gu et al., are gaining popularity as effective approaches for modeling
sequential data. Deep SSMs demonstrate outstanding performance across a diverse
set of domains, at a reduced training and inference cost compared to
attention-based transformers. Recent developments show that if the linear
recurrence powering SSMs allows for multiplicative interactions between inputs
and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture
can surpass in both in accuracy and efficiency attention-powered foundation
models trained on text, at scales of billion parameters. In this paper, we give
theoretical grounding to this recent finding using tools from Rough Path
Theory: we show that when random linear recurrences are equipped with simple
input-controlled transitions (selectivity mechanism), then the hidden state is
provably a low-dimensional projection of a powerful mathematical object called
the signature of the input -- capturing non-linear interactions between tokens
at distinct timescales. Our theory not only motivates the success of modern
selective state-space models such as Mamba but also provides a solid framework
to understand the expressive power of future SSM variants.

基于 Rough Path Theory，本研究证明了具备选择性机制的随机线性递归在输入控制条件下可产生低维投射的隐藏状态，并阐述了现代选择性状态空间模型的成功和未来 SSM 变体的表达能力。

深度选择状态空间模型的理论基础

Theoretical Foundations of Deep Selective State-Space Models

Linear Recurrence has proven to be a powerful tool for modeling long
sequences efficiently. In this work, we show that existing models fail to take
full advantage of its potential. Motivated by this finding, we develop
GateLoop, a foundational sequence model that generalizes linear recurrent
models such as S4, S5, LRU and RetNet, by employing data-controlled state
transitions. Utilizing this theoretical advance, GateLoop empirically
outperforms existing models for auto-regressive language modeling. Our method
comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \log_{2} l)$
parallel mode making use of highly optimized associative scan implementations.
Furthermore, we derive an $O(l^2)$ surrogate attention mode, revealing
remarkable implications for Transformer and recently proposed architectures.
Specifically, we prove that our approach can be interpreted as providing
data-controlled relative-positional information to Attention. While many
existing models solely rely on data-controlled cumulative sums for context
aggregation, our findings suggest that incorporating data-controlled complex
cumulative products may be a crucial step towards more powerful sequence
models.

门控循环模型 GateLoop 利用了线性递推模型的潜力，通过采用数据控制的状态转换，推广了线性递归模型，如 S4、S5、LRU 和 RetNet，实现了比现有模型更优越的自回归语言建模，具有低成本的递归模式和高效的并行模式以及对 Transformer 和最近提出的架构的重要影响。同时揭示了数据控制的相对位置信息对注意力机制的意义，并建议了将数据控制的复杂累积乘积纳入上下文聚合的关键步骤，以实现更强大的序列模型。