In recent studies, line search methods have shown significant improvements in
the performance of traditional stochastic gradient descent techniques,
eliminating the need for a specific learning rate schedule. In this paper, we
identify existing issues in state-of-the-art line search methods, propose
enhancements, and rigorously evaluate their effectiveness. We test these
methods on larger datasets and more complex data domains than before.
Specifically, we improve the Armijo line search by integrating the momentum
term from ADAM in its search direction, enabling efficient large-scale
training, a task that was previously prone to failure using Armijo line search
methods. Our optimization approach outperforms both the previous Armijo
implementation and tuned learning rate schedules for Adam. Our evaluation
focuses on Transformers and CNNs in the domains of NLP and image data. Our work
is publicly available as a Python package, which provides a hyperparameter free
Pytorch optimizer.

使用线搜索方法改进了传统随机梯度下降技术，通过在搜索方向中整合 ADAM 的动量项，实现了高效的大规模训练，提高了性能。

优化大规模神经网络训练的线搜索方法

Improving Line Search Methods for Large Scale Neural Network Training

Recent works have shown that line search methods greatly increase performance
of traditional stochastic gradient descent methods on a variety of datasets and
architectures [1], [2]. In this work we succeed in extending line search
methods to the novel and highly popular Transformer architecture and dataset
domains in natural language processing. More specifically, we combine the
Armijo line search with the Adam optimizer and extend it by subdividing the
networks architecture into sensible units and perform the line search
separately on these local units. Our optimization method outperforms the
traditional Adam optimizer and achieves significant performance improvements
for small data sets or small training budgets, while performing equal or better
for other tested cases. Our work is publicly available as a python package,
which provides a hyperparameter-free pytorch optimizer that is compatible with
arbitrary network architectures.

本研究成功将线搜索方法应用于新颖且受欢迎的 Transformer 架构和自然语言处理数据领域，通过将网络架构细分为合理的单元并在这些局部单元上进行线搜索，将 Armijo 线搜索与 Adam 优化器结合，其优化方法在小数据集或小训练预算的情况下超越传统的 Adam 优化器，并在其他测试案例中表现相等甚至更好。我们的工作以 Python 包的形式公开，提供一个与任意网络架构兼容且无需超参数的 PyTorch 优化器。

使用线搜索方法加速 Transformer 微调的收敛

Faster Convergence for Transformer Fine-tuning with Line Search Methods

Stochastic gradient descent (SGD) is the simplest deep learning optimizer
with which to train deep neural networks. While SGD can use various learning
rates, such as constant or diminishing rates, the previous numerical results
showed that SGD performs better than other deep learning optimizers using when
it uses learning rates given by line search methods. In this paper, we perform
a convergence analysis on SGD with a learning rate given by an Armijo line
search for nonconvex optimization. The analysis indicates that the upper bound
of the expectation of the squared norm of the full gradient becomes small when
the number of steps and the batch size are large. Next, we show that, for SGD
with the Armijo-line-search learning rate, the number of steps needed for
nonconvex optimization is a monotone decreasing convex function of the batch
size; that is, the number of steps needed for nonconvex optimization decreases
as the batch size increases. Furthermore, we show that the stochastic
first-order oracle (SFO) complexity, which is the stochastic gradient
computation cost, is a convex function of the batch size; that is, there exists
a critical batch size that minimizes the SFO complexity. Finally, we provide
numerical results that support our theoretical results. The numerical results
indicate that the number of steps needed for training deep neural networks
decreases as the batch size increases and that there exist the critical batch
sizes that can be estimated from the theoretical results.

随机梯度下降是最简单的深度学习优化器之一，该论文通过蒙特卡洛方法对其进行了收敛性分析，并证明了使用 Armijo 线搜索的随机梯度下降在非凸优化中的性能优于其他深度学习优化器，同时还发现了批量大小对训练的影响，批量大小越大，需要的步数越少，但在成本和梯度计算的角度，存在一个临界批量大小最能降低成本。