We introduce "AnnealSGD", a regularized stochastic gradient descent algorithm
motivated by an analysis of the energy landscape of a particular class of deep
networks with sparse random weights. The loss function of such networks can be
approximated by the Hamiltonian of a spherical spin glass with Gaussian
coupling. While different from currently-popular architectures such as
convolutional ones, spin glasses are amenable to analysis, which provides
insights on the topology of the loss function and motivates algorithms to
minimize it. Specifically, we show that a regularization term akin to a
magnetic field can be modulated with a single scalar parameter to transition
the loss function from a complex, non-convex landscape with exponentially many
local minima, to a phase with a polynomial number of minima, all the way down
to a trivial landscape with a unique minimum. AnnealSGD starts training in the
relaxed polynomial regime and gradually tightens the regularization parameter
to steer the energy towards the original exponential regime. Even for
convolutional neural networks, which are quite unlike sparse random networks,
we empirically show that AnnealSGD improves the generalization error using
competitive baselines on MNIST and CIFAR-10.

本文提出了一种名为 AnnealSGD 的正则化随机梯度下降算法，可以通过对一个特定类别的深度网络的能量景观进行分析来获得启示，从而优化损失函数。