Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such "out-of-equilibrium" behavior is a consequence of the fact that the gradient noise in SGD is highly non-isotropic; the covariance matrix of mini-batch gradients has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.

SGD使用隐式正则化训练深度神经网络的确切方式一直以来都很难界定，但我们证明了SGD最小化了一个与分布式权重后验分布相关的平均势能加一个熵正则化项，但这个势能通常与原始损失函数不同。此外，我们表明SGD在经典意义下甚至不会收敛，因此这种'失衡'的行为是由于对于深度网络，SGD的梯度噪声是高度非各向同性的。我们在附录中给出了这些声明的广泛经验验证。

随机梯度下降在变分推断中的应用：深度网络收敛于极限环