In recurrent neural networks, learning long-term dependency is the main difficulty due to the vanishing and exploding gradient problem. Many researchers are dedicated to solving this issue and they proposed many algorithms. Although these algorithms have achieved great success, understanding how the information decays remains an open problem. In this paper, we study the dynamics of the hidden state in recurrent neural networks. We propose a new perspective to analyze the hidden state space based on an eigen decomposition of the weight matrix. We start the analysis by linear state space model and explain the function of preserving information in activation functions. We provide an explanation for long-term dependency based on the eigen analysis. We also point out the different behavior of eigenvalues for regression tasks and classification tasks. From the observations on well-trained recurrent neural networks, we proposed a new initialization method for recurrent neural networks, which improves consistently performance. It can be applied to vanilla-RNN, LSTM, and GRU. We test on many datasets, such as Tomita Grammars, pixel-by-pixel MNIST datasets, and machine translation datasets (Multi30k). It outperforms the Xavier initializer and kaiming initializer as well as other RNN-only initializers like IRNN and sp-RNN in several tasks.

在本文中，我们通过权重矩阵的特征分解提出了一种分析循环神经网络中隐藏状态空间的新视角，解释了激活函数中的信息保存功能，基于特征分析为长期依赖提供了一个解释，并指出了回归任务和分类任务中特征值的不同行为。通过对训练良好的循环神经网络进行观察，我们提出了一种改良性能的新的循环神经网络初始化方法，在多个任务上胜过Xavier初始化器、kaiming初始化器以及其他仅适用于循环神经网络的初始化器，如IRNN和sp-RNN。

递归神经网络的动态分析和特征值初始化器