We prove an inverse approximation theorem for the approximation of nonlinear
sequence-to-sequence relationships using RNNs. This is a so-called
Bernstein-type result in approximation theory, which deduces properties of a
target function under the assumption that it can be effectively approximated by
a hypothesis space. In particular, we show that nonlinear sequence
relationships, viewed as functional sequences, that can be stably approximated
by RNNs with hardtanh/tanh activations must have an exponential decaying memory
structure -- a notion that can be made precise. This extends the previously
identified curse of memory in linear RNNs into the general nonlinear setting,
and quantifies the essential limitations of the RNN architecture for learning
sequential relationships with long-term memory. Based on the analysis, we
propose a principled reparameterization method to overcome the limitations. Our
theoretical results are confirmed by numerical experiments.

该研究提出了一种逆逼近定理，论证了使用 RNN 对非线性序列关系进行逼近的能力受到存储结构的限制，并提出了一种基于理论分析的重新参数化方法以克服这种限制。

非线性递归神经网络的逆近似理论

Inverse Approximation Theory for Nonlinear Recurrent Neural Networks

We present a theoretical analysis of the approximation properties of
convolutional architectures when applied to the modeling of temporal sequences.
Specifically, we prove an approximation rate estimate (Jackson-type result) and
an inverse approximation theorem (Bernstein-type result), which together
provide a comprehensive characterization of the types of sequential
relationships that can be efficiently captured by a temporal convolutional
architecture. The rate estimate improves upon a previous result via the
introduction of a refined complexity measure, whereas the inverse approximation
theorem is new.

本篇论文对卷积架构在建模时序序列中的逼近性质进行了理论分析，证明了逼近速率估计和逆逼近定理，并通过引入改进的复杂度测度来提高速率估计。逆逼近定理是新的，两者共同提供了卷积架构能够高效捕捉的时序关系类型的综合特征。