Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike a recurrent network, where the same matrix is applied repeatedly, an FFN applies a different matrix at each layer. We show that the successive application of correctly scaled random matrices to an initial vector results in a random walk of the log of the norm of the resulting vectors. This random walk has zero mean and variance that is linear in the number of matrix applications. The variance is also inversely proportional to the dimension of the vector. We apply these random matrix results to very deep FFNs with hundreds of layers by observing that the backprop equations are linear in the errors. Practically, this implies a vanishing gradient whose log-norm scales with the square root of the network depth. It further implies that the vanishing gradient problem can be mitigated by increasing the width of the layers. Mathematical analyses and experimental results using stochastic gradient descent to optimize the MNIST and TIMIT datasets are provided to support these claims.

该研究论文探讨了在机器学习中训练深度网络的困难之处，并提出了一种方法解决梯度消失问题，即适当增加各层的宽度以缓解问题。

用随机游走初始化训练超深度前馈网络