We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability.

本研究通过实验证明神经网络训练目标的全批量梯度下降通常处于稳定性的边缘状态。在这种状态下，训练损失Hessian的最大特征值略高于数值$2/	ext{(步长)}$，训练损失在短时间内呈现非单调行为，但在长时间尺度上保持下降态势。鉴于这种行为与优化领域中的一些传统观念不一致，我们的发现提出了关于这些观念是否与神经网络训练 relevant 的质疑。我们希望我们的研究能够激发未来针对稳定性边缘优化问题的进一步研究。

神经网络梯度下降通常发生在稳定边缘