Gradient descent and its variants are de facto standard algorithms for
training machine learning models. As gradient descent is sensitive to its
hyperparameters, we need to tune the hyperparameters carefully using a grid
search, but it is time-consuming, especially when multiple hyperparameters
exist. Recently, parameter-free methods that adjust the hyperparameters on the
fly have been studied. However, the existing work only studied parameter-free
methods for the stepsize, and parameter-free methods for other hyperparameters
have not been explored. For instance, the gradient clipping threshold is also a
crucial hyperparameter in addition to the stepsize to prevent gradient
explosion issues, but none of the existing studies investigated the
parameter-free methods for clipped gradient descent. In this work, we study the
parameter-free methods for clipped gradient descent. Specifically, we propose
Inexact Polyak Stepsize, which converges to the optimal solution without any
hyperparameters tuning, and its convergence rate is asymptotically independent
of L under L-smooth and $(L_0, L_1)$-smooth assumptions of the loss function as
that of clipped gradient descent with well-tuned hyperparameters. We
numerically validated our convergence results using a synthetic function and
demonstrated the effectiveness of our proposed methods using LSTM, Nano-GPT,
and T5.

本研究探讨了参数自由方法在修剪的梯度下降中的应用，提出了非精确 Polyak 步长方法，其收敛速度与拥有良好调整的超参数的修剪梯度下降方法在 L 平滑和（L0，L1）平滑假设下，收敛到最优解的速度渐近独立。在合成函数和 LSTM、Nano-GPT 和 T5 的实验中验证了我们的收敛结果和方法的有效性。