Curvature information -- particularly, the largest eigenvalue of the loss Hessian, known as the sharpness -- often forms the basis for learning rate tuners. However, recent work has shown that the curvature information undergoes complex dynamics during training, going from a phase of increasing sharpness to eventual stabilization. We analyze the closed-loop feedback effect between learning rate tuning and curvature. We find that classical learning rate tuners may yield greater one-step loss reduction, yet they ultimately underperform in the long term when compared to constant learning rates in the full batch regime. These models break the stabilization of the sharpness, which we explain using a simplified model of the joint dynamics of the learning rate and the curvature. To further investigate these effects, we introduce a new learning rate tuning method, Curvature Dynamics Aware Tuning (CDAT), which prioritizes long term curvature stabilization over instantaneous progress on the objective. In the full batch regime, CDAT shows behavior akin to prefixed warm-up schedules on deep learning objectives, outperforming tuned constant learning rates. In the mini batch regime, we observe that stochasticity introduces confounding effects that explain the previous success of some learning rate tuners at appropriate batch sizes. Our findings highlight the critical role of understanding the joint dynamics of the learning rate and curvature, beyond greedy minimization, to diagnose failures and design effective adaptive learning rate tuners.

通过分析学习率调整和曲率间的闭环反馈效应，我们发现传统的学习率调整器在满批处理模式下表现不佳，无法稳定曲率，因此我们提出了一种新的学习率调整方法，称为曲率动态感知调整（CDAT），它优先考虑目标的长期曲率稳定性。在深度学习目标上，CDAT的表现类似于预定的预热调度，优于调整后的恒定学习率。在小批次处理模式下，我们观察到随机性引入了混淆效应，解释了一些学习率调整器在适当的批次大小上取得成功的原因。我们的研究强调了理解学习率和曲率的联合动态的关键作用，超越贪婪最小化，在诊断故障和设计有效的自适应学习率调整器方面具有重要意义。

曲率感知学习率调整器