State-space models (SSMs) have recently emerged as a framework for learning
long-range sequence tasks. An example is the structured state-space sequence
(S4) layer, which uses the diagonal-plus-low-rank structure of the HiPPO
initialization framework. However, the complicated structure of the S4 layer
poses challenges; and, in an effort to address these challenges, models such as
S4D and S5 have considered a purely diagonal structure. This choice simplifies
the implementation, improves computational efficiency, and allows channel
communication. However, diagonalizing the HiPPO framework is itself an
ill-posed problem. In this paper, we propose a general solution for this and
related ill-posed diagonalization problems in machine learning. We introduce a
generic, backward-stable "perturb-then-diagonalize" (PTD) methodology, which is
based on the pseudospectral theory of non-normal operators, and which may be
interpreted as the approximate diagonalization of the non-normal matrices
defining SSMs. Based on this, we introduce the S4-PTD and S5-PTD models.
Through theoretical analysis of the transfer functions of different
initialization schemes, we demonstrate that the S4-PTD/S5-PTD initialization
strongly converges to the HiPPO framework, while the S4D/S5 initialization only
achieves weak convergences. As a result, our new models show resilience to
Fourier-mode noise-perturbed inputs, a crucial property not achieved by the
S4D/S5 models. In addition to improved robustness, our S5-PTD model averages
87.6% accuracy on the Long-Range Arena benchmark, demonstrating that the PTD
methodology helps to improve the accuracy of deep learning models.

在这篇论文中，我们提出了一个用于解决机器学习中的病态对角化问题的通用、稳定的 “扰动 - 然后对角化” (PTD) 方法，并基于此方法引入了 S4-PTD 和 S5-PTD 模型。通过对不同初始化方案的传递函数进行理论分析，我们证明了 S4-PTD/S5-PTD 初始化可以强大地收敛于 HiPPO 框架，而 S4D/S5 初始化只能实现弱收敛。因此，我们的新模型对傅里叶模式噪声扰动输入表现出了鲁棒性，这是 S4D/S5 模型所无法达到的重要特性。此外，我们的 S5-PTD 模型在 Long-Range Arena 基准上平均达到 87.6% 的准确率，证明了 PTD 方法在提升深度学习模型的准确性方面起到了积极作用。