Hyperparameter tuning, particularly the selection of an appropriate learning
rate in adaptive gradient training methods, remains a challenge. To tackle this
challenge, in this paper, we propose a novel parameter-free optimizer, AdamG
(Adam with the golden step size), designed to automatically adapt to diverse
optimization problems without manual tuning. The core technique underlying
AdamG is our golden step size derived for the AdaGrad-Norm algorithm, which is
expected to help AdaGrad-Norm preserve the tuning-free convergence and
approximate the optimal step size in expectation w.r.t. various optimization
scenarios. To better evaluate tuning-free performance, we propose a novel
evaluation criterion, stability, to comprehensively assess the efficacy of
parameter-free optimizers in addition to classical performance criteria.
Empirical results demonstrate that compared with other parameter-free
baselines, AdamG achieves superior performance, which is consistently on par
with Adam using a manually tuned learning rate across various optimization
tasks.

提出了一种无需手动调节参数的优化器 AdamG，通过使用 AdaGrad-Norm 算法中的黄金步长派生技术，自动适应不同的优化问题并实现了与手动调节学习率的 Adam 相当的优化性能。