BriefGPT.xyz
Jul, 2023
轨迹对齐:通过分岔理论理解稳定边缘现象
Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory
HTML
PDF
Minhak Song, Chulhee Yun
TL;DR
通过实证研究,证明最大特征值(也被称为锐度)沿着梯度下降轨迹的演化呈现出一种叫做稳定边缘现象(EoS)的现象,进一步证明了在合适的重新参数化下,不同的梯度下降轨迹会在一个特定的分叉图上对齐,从而建立了锐度逐步增加和EoS现象的理论分析。
Abstract
Cohen et al. (2021) empirically study the
evolution
of the largest eigenvalue of the
loss hessian
, also known as
sharpness
, along the grad
→