Deep neural networks (DNNs) have been successfully applied to many real-world
problems, but a complete understanding of their dynamical and computational
principles is still lacking. Conventional theoretical frameworks for analysing
DNNs often assume random networks with coupling weights obeying Gaussian
statistics. However, non-Gaussian, heavy-tailed coupling is a ubiquitous
phenomenon in DNNs. Here, by weaving together theories of heavy-tailed random
matrices and non-equilibrium statistical physics, we develop a new type of mean
field theory for DNNs which predicts that heavy-tailed weights enable the
emergence of an extended critical regime without fine-tuning parameters. In
this extended critical regime, DNNs exhibit rich and complex propagation
dynamics across layers. We further elucidate that the extended criticality
endows DNNs with profound computational advantages: balancing the contraction
as well as expansion of internal neural representations and speeding up
training processes, hence providing a theoretical guide for the design of
efficient neural architectures.

利用长尾随机矩阵与非平衡统计力学理论，提出了 DNN 的新型平均场理论，并发现重尾权重使得 DNN 出现了一个拓展的临界区，体现了丰富的跨层传播动力学，进而赋予 DNN 突出的计算优势，这为设计高效神经网络结构提供了理论指导。