Orthogonal parameterization is a compelling solution to the vanishing
gradient problem (VGP) in recurrent neural networks (RNNs). With orthogonal
parameters and non-saturated activation functions, gradients in such models are
constrained to unit norms. On the other hand, although the traditional vanilla
RNNs are seen to have higher memory capacity, they suffer from the VGP and
perform badly in many applications. This work proposes Adaptive-Saturated RNNs
(asRNN), a variant that dynamically adjusts its saturation level between the
two mentioned approaches. Consequently, asRNN enjoys both the capacity of a
vanilla RNN and the training stability of orthogonal RNNs. Our experiments show
encouraging results of asRNN on challenging sequence learning benchmarks
compared to several strong competitors. The research code is accessible at
this https URL

本文研究借助正交参数化技术解决循环神经网络中的梯度弥散问题，提出了一种 Adaptive-Saturated RNNs (ASRNN) 变种，它动态调整饱和度水平，融合传统循环神经网络的高内存容量与正交循环神经网络的稳定性。实验结果表明，ASRNN 比其他几种竞争对手在具有挑战性的序列学习基准上表现更好。