Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep
Neural Networks (DNNs), including both production quality, pre-trained models
such as AlexNet and Inception, and smaller models trained from scratch, such as
LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly
indicate that the empirical spectral density (ESD) of DNN layer matrices
displays signatures of traditionally-regularized statistical models, even in
the absence of exogenously specifying traditional forms of regularization, such
as Dropout or Weight Norm constraints. Building on recent results in RMT, most
notably its extension to Universality classes of Heavy-Tailed matrices, we
develop a theory to identify \emph{5+1 Phases of Training}, corresponding to
increasing amounts of \emph{Implicit Self-Regularization}. For smaller and/or
older DNNs, this Implicit Self-Regularization is like traditional Tikhonov
regularization, in that there is a `size scale' separating signal from noise.
For state-of-the-art DNNs, however, we identify a novel form of
\emph{Heavy-Tailed Self-Regularization}, similar to the self-organization seen
in the statistical physics of disordered systems. This implicit
Self-Regularization can depend strongly on the many knobs of the training
process. By exploiting the generalization gap phenomena, we demonstrate that we
can cause a small model to exhibit all 5+1 phases of training simply by
changing the batch size.

本研究运用随机矩阵理论分析了深度神经网络的权重矩阵，通过实验和理论结果表明神经网络层矩阵的经验谱密度显示出传统正则化统计模型的特征。并且发现随着训练阶段的增加，隐式自正则化逐渐显现，这种隐式自正则化可以像传统的 Tikhonov 正则化一样，但也可以是重尾分布的，类似于无序系统的自组织现象，并存在着一个通用性的现象。