Adaptive optimization algorithms such as Adam are widely used in deep
learning. The stability of such algorithms is often improved with a warmup
schedule for the learning rate. Motivated by the difficulty of choosing and
tuning warmup schedules, recent work proposes automatic variance rectification
of Adam's adaptive learning rate, claiming that this rectified approach
("RAdam") surpasses the vanilla Adam algorithm and reduces the need for
expensive tuning of Adam with warmup. In this work, we refute this analysis and
provide an alternative explanation for the necessity of warmup based on the
magnitude of the update term, which is of greater relevance to training
stability. We then provide some "rule-of-thumb" warmup schedules, and we
demonstrate that simple untuned warmup of Adam performs more-or-less
identically to RAdam in typical practical settings. We conclude by suggesting
that practitioners stick to linear warmup with Adam, with a sensible default
being linear warmup over $2 / (1 - \beta_2)$ training iterations.

本研究分析 Adam 的自适应学习率，提出温升调整学习率的必要性取决于更新项的大小，进而提供一些温升规则，并表明未调整的 Adam 与 RAdam 在典型实际环境中表现基本相同，建议从业者在 Adam 中坚持使用线性温升。