There currently is a significant interest in understanding the Edge of Stability (EoS) phenomenon, which has been observed in neural networks training, characterized by a non-monotonic decrease of the loss function over epochs, while the sharpness of the loss (spectral norm of the Hessian) progressively approaches and stabilizes around 2/(learning rate). Reasons for the existence of EoS when training using gradient descent have recently been proposed -- a lack of flat minima near the gradient descent trajectory together with the presence of compact forward-invariant sets. In this paper, we show that linear neural networks optimized under a quadratic loss function satisfy the first assumption and also a necessary condition for the second assumption. More precisely, we prove that the gradient descent map is non-singular, the set of global minimizers of the loss function forms a smooth manifold, and the stable minima form a bounded subset in parameter space. Additionally, we prove that if the step-size is too big, then the set of initializations from which gradient descent converges to a critical point has measure zero.

在本文中，我们证明了在使用二次损失函数优化的线性神经网络中，梯度下降映射是非奇异的，损失函数的全局极小化集合形成平滑流形，并且稳定的极小值在参数空间中形成有界子集。另外，我们证明了如果步长过大，则使梯度下降收敛到临界点的初始化集合的测度为零。

大学习速率下梯度下降的稳定性