Deep neural networks based on state space models (SSMs) are attracting much
attention in sequence modeling since their computational cost is significantly
smaller than that of Transformers. While the capabilities of SSMs have been
primarily investigated through experimental comparisons, theoretical
understanding of SSMs is still limited. In particular, there is a lack of
statistical and quantitative evaluation of whether SSM can replace
Transformers. In this paper, we theoretically explore in which tasks SSMs can
be alternatives of Transformers from the perspective of estimating
sequence-to-sequence functions. We consider the setting where the target
function has direction-dependent smoothness and prove that SSMs can estimate
such functions with the same convergence rate as Transformers. Additionally, we
prove that SSMs can estimate the target function, even if the smoothness
changes depending on the input sequence, as well as Transformers. Our results
show the possibility that SSMs can replace Transformers when estimating the
functions in certain classes that appear in practice.

基于状态空间模型的深度神经网络在序列建模中引起了广泛关注，本文从估计序列到序列函数的角度出发，理论探讨了状态空间模型可以替代 Transformers 的任务范围，并证明了状态空间模型能够在特定类别的函数估计中取代 Transformers。