When using large-batch training to speed up stochastic gradient descent,
learning rates must adapt to new batch sizes in order to maximize speed-ups and
preserve model quality. Re-tuning learning rates is resource intensive, while
fixed scaling rules often degrade model quality. We propose AdaScale SGD, an
algorithm that reliably adapts learning rates to large-batch training. By
continually adapting to the gradient's variance, AdaScale automatically
achieves speed-ups for a wide range of batch sizes. We formally describe this
quality with AdaScale's convergence bound, which maintains final objective
values, even as batch sizes grow large and the number of iterations decreases.
In empirical comparisons, AdaScale trains well beyond the batch size limits of
popular "linear learning rate scaling" rules. This includes large-batch
training with no model degradation for machine translation, image
classification, object detection, and speech recognition tasks. AdaScale's
qualitative behavior is similar to that of "warm-up" heuristics, but unlike
warm-up, this behavior emerges naturally from a principled mechanism. The
algorithm introduces negligible computational overhead and no new
hyperparameters, making AdaScale an attractive choice for large-scale training
in practice.

AdaScale SGD 是一种可靠地适应大批量训练学习率的算法，通过不断调整梯度方差，可以实现在宽范围的批量大小下加速训练。在机器翻译，图像分类，目标检测和语音识别任务中，它可以处理大批量训练，而不会降低模型质量，因为 AdaScale 的收敛性边界可以保持最终目标值，即使批量大小增加，迭代次数减少。这个算法引入了可忽略的计算开销和没有新的超参数，因此是实现大规模训练的有吸引力的选择。