In a neural network (NN), *weight matrices* linearly transform inputs into
*preactivations* that are then transformed nonlinearly into *activations*. A
typical NN interleaves multitudes of such linear and nonlinear transforms to
express complex functions. Thus, the (pre-)activations depend on the weights in
an intricate manner. We show that, surprisingly, (pre-)activations of a
randomly initialized NN become *independent* from the weights as the NN's
widths tend to infinity, in the sense of asymptotic freeness in random matrix
theory. We call this the Free Independence Principle (FIP), which has these
consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian
singular value distribution of an NN in Pennington et al. [36,37], essential
for training ultra-deep NNs [48]. 2) It gives a new justification of gradient
independence assumption used for calculating the Neural Tangent Kernel of a
neural network. FIP and these results hold for any neural architecture. We show
FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang
[50,51], generalizing the Master Theorems proved in those works. As warmup
demonstrations of this new Master Theorem, we give new proofs of the semicircle
and Marchenko-Pastur laws, which benchmarks our framework against these
fundamental mathematical results.

该论文发现神经网络中的权重与激活函数节点随着神经网络的宽度趋近于无穷时变得独立，并且给出了这个发现的几个相关应用。