Learning rate schedules used in practice bear little resemblance to those
recommended by theory. We close much of this theory/practice gap, and as a
consequence are able to derive new problem-adaptive learning rate schedules.
Our key technical contribution is a refined analysis of learning rate schedules
for a wide class of optimization algorithms (including SGD). In contrast to
most prior works that study the convergence of the average iterate, we study
the last iterate, which is what most people use in practice. When considering
only worst-case analysis, our theory predicts that the best choice is the
linear decay schedule: a popular choice in practice that sets the stepsize
proportionally to $1 - t/T$, where $t$ is the current iteration and $T$ is the
total number of steps. To go beyond this worst-case analysis, we use the
observed gradient norms to derive schedules refined for any particular task.
These refined schedules exhibit learning rate warm-up and rapid learning rate
annealing near the end of training. Ours is the first systematic approach to
automatically yield both of these properties. We perform the most comprehensive
evaluation of learning rate schedules to date, evaluating across 10 diverse
deep learning problems, a series of LLMs, and a suite of logistic regression
problems. We validate that overall, the linear-decay schedule matches or
outperforms all commonly used default schedules including cosine annealing, and
that our schedule refinement method gives further improvements.

学习率调度与优化算法的收敛性分析、学习率预热和调度优化方法的研究。

何时、为何以及何倍？通过优化改进的自适应学习率调度

When, Why and How Much? Adaptive Learning Rate Scheduling by Refinement

The Transformer is widely used in natural language processing tasks. To train
a Transformer however, one usually needs a carefully designed learning rate
warm-up stage, which is shown to be crucial to the final performance but will
slow down the optimization and bring more hyper-parameter tunings. In this
paper, we first study theoretically why the learning rate warm-up stage is
essential and show that the location of layer normalization matters.
Specifically, we prove with mean field theory that at initialization, for the
original-designed Post-LN Transformer, which places the layer normalization
between the residual blocks, the expected gradients of the parameters near the
output layer are large. Therefore, using a large learning rate on those
gradients makes the training unstable. The warm-up stage is practically helpful
for avoiding this problem. On the other hand, our theory also shows that if the
layer normalization is put inside the residual blocks (recently proposed as
Pre-LN Transformer), the gradients are well-behaved at initialization. This
motivates us to remove the warm-up stage for the training of Pre-LN
Transformers. We show in our experiments that Pre-LN Transformers without the
warm-up stage can reach comparable results with baselines while requiring
significantly less training time and hyper-parameter tuning on a wide range of
applications.

本文证明了学习率预热阶段对于 Post-LN Transformer 是必要的，并提出了一种新的 Pre-LN Transformer 结构，该结构无需预热阶段，同时具有与基线相当的性能，大大减少了训练时间和超参数调整。