Recent research shows that when Gradient Descent (GD) is applied to neural networks, the loss almost never decreases monotonically. Instead, the loss oscillates as gradient descent converges to its ''Edge of Stability'' (EoS). Here, we find a quantity that does decrease monotonically throughout GD training: the sharpness attained by the gradient flow solution (GFS)-the solution that would be obtained if, from now until convergence, we train with an infinitesimal step size. Theoretically, we analyze scalar neural networks with the squared loss, perhaps the simplest setting where the EoS phenomena still occur. In this model, we prove that the GFS sharpness decreases monotonically. Using this result, we characterize settings where GD provably converges to the EoS in scalar networks. Empirically, we show that GD monotonically decreases the GFS sharpness in a squared regression model as well as practical neural network architectures.

应用梯度下降 (GD) 到神经网络时，损失函数几乎从不呈单调递减。我们找到了一种随着 GD 训练而单调递减的量：梯度流解 (GFS) 所达到的锐度。在理论上，我们分析了具有平方损失的标量神经网络，这可能是出现 EoS 现象最简单的设置。我们在模型中证明了 GFS 锐度单调递减的结果，并表征了在标量网络中 GD 可以证明收敛到 EoS 的设置。从经验上看，我们展示了在平方回归模型以及实用的神经网络架构中，GD 单调递减 GFS 锐度。

梯度下降单调减小标量网络及其他解的梯度流锐度