The infinitely wide neural network has been proven a useful and manageable
mathematical model that enables the understanding of many phenomena appearing
in deep learning. One example is the convergence of random deep networks to
Gaussian processes that allows a rigorous analysis of the way the choice of
activation function and network weights impacts the training dynamics. In this
paper, we extend the seminal proof of Matthews et al. (2018) to a larger class
of initial weight distributions (which we call PSEUDO-IID), including the
established cases of IID and orthogonal weights, as well as the emerging
low-rank and structured sparse settings celebrated for their computational
speed-up benefits. We show that fully-connected and convolutional networks
initialized with PSEUDO-IID distributions are all effectively equivalent up to
their variance. Using our results, one can identify the Edge-of-Chaos for a
broader class of neural networks and tune them at criticality in order to
enhance their training.

本文扩展了 Matthews 等人 (2018) 的重要证明，针对一类更大的初始权重分布（我们称之为 PSEUDO-IID），包括 IID 和正交权重，以及计算速度加快的低秩和结构稀疏设置。我们展示了使用 PSEUDO-IID 分布初始化的全连接和卷积网络在方差上是等价的，通过这些结果，可以为更广泛的神经网络识别边缘混沌，并调整它们以提高训练效果。