We introduce a novel dynamic learning-rate scheduling scheme grounded in
theory with the goal of simplifying the manual and time-consuming tuning of
schedules in practice. Our approach is based on estimating the locally-optimal
stepsize, guaranteeing maximal descent in the direction of the stochastic
gradient of the current step. We first establish theoretical convergence bounds
for our method within the context of smooth non-convex stochastic optimization,
matching state-of-the-art bounds while only assuming knowledge of the
smoothness parameter. We then present a practical implementation of our
algorithm and conduct systematic experiments across diverse datasets and
optimization algorithms, comparing our scheme with existing state-of-the-art
learning-rate schedulers. Our findings indicate that our method needs minimal
tuning when compared to existing approaches, removing the need for auxiliary
manual schedules and warm-up phases and achieving comparable performance with
drastically reduced parameter tuning.

我们介绍了一种新颖的动态学习率调度方案，旨在简化实践中手动而耗时的调度。我们的方法基于估计局部最优步长，确保在当前步骤的随机梯度方向上实现最大下降。我们首先在平滑的非凸随机优化环境中建立了我们方法的理论收敛边界，与最新的边界相匹配，仅假设对平滑参数具有知识。然后，我们提出了我们算法的实际实现，并在不同数据集和优化算法上进行系统实验，将我们的方案与现有的最新学习率调度器进行比较。我们的发现表明，与现有方法相比，我们的方法需要最少的调整，消除了辅助手动调度和预热阶段的需要，并实现了具有大大减少参数调整的可比性能。