The stochastic gradient descent (SGD) algorithm is the algorithm we use to
train neural networks. However, it remains poorly understood how the SGD
navigates the highly nonlinear and degenerate loss landscape of a neural
network. In this work, we prove that the minibatch noise of SGD regularizes the
solution towards a balanced solution whenever the loss function contains a
rescaling symmetry. Because the difference between a simple diffusion process
and SGD dynamics is the most significant when symmetries are present, our
theory implies that the loss function symmetries constitute an essential probe
of how SGD works. We then apply this result to derive the stationary
distribution of stochastic gradient flow for a diagonal linear network with
arbitrary depth and width. The stationary distribution exhibits complicated
nonlinear phenomena such as phase transitions, broken ergodicity, and
fluctuation inversion. These phenomena are shown to exist uniquely in deep
networks, implying a fundamental difference between deep and shallow models.

随机梯度下降（SGD）算法是用于训练神经网络的算法。在这项工作中，我们证明了 SGD 的小批量噪声在有缩放对称性的损失函数中使解决方案朝着均衡解决方案正则化。我们应用这一结果来推导具有任意深度和宽度的对角线线性网络的随机梯度流的稳态分布，该稳态分布展示了相变、破坏性遍历和涨落反转等复杂的非线性现象，这些现象只存在于深度网络中，暗示着深度模型与浅模型之间的根本差异。