Deep neural networks exhibit a fascinating spectrum of phenomena ranging from
predictable scaling laws to the unpredictable emergence of new capabilities as
a function of training time, dataset size and network size. Analysis of these
phenomena has revealed the existence of concepts and algorithms encoded within
the learned representations of these networks. While significant strides have
been made in explaining observed phenomena separately, a unified framework for
understanding, dissecting, and predicting the performance of neural networks is
lacking. Here, we introduce the semantic landscape paradigm, a conceptual and
mathematical framework that describes the training dynamics of neural networks
as trajectories on a graph whose nodes correspond to emergent algorithms that
are instrinsic to the learned representations of the networks. This abstraction
enables us to describe a wide range of neural network phenomena in terms of
well studied problems in statistical physics. Specifically, we show that
grokking and emergence with scale are associated with percolation phenomena,
and neural scaling laws are explainable in terms of the statistics of random
walks on graphs. Finally, we discuss how the semantic landscape paradigm
complements existing theoretical and practical approaches aimed at
understanding and interpreting deep neural networks.

深度神经网络的训练动态可以用语义景观范式来描述，该范式将神经网络的学习表示抽象为图上的轨迹，并解释了与尺度相关的算法的出现以及神经尺度定律的统计物理学现象。