The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: https://github.com/LiyuanLucasLiu/RAdam.

文中探讨学习率预热方法在稳定训练、加速收敛和改善通用性方面的可靠性，发现自适应学习率在初始阶段有问题，建议使用预热作为方差缩减技术，并提出了一种新变量RAdam用于改善自适应学习率方差，实验结果表明其有效性和鲁棒性。

自适应学习率的方差及更多