Large-batch training has been essential in leveraging large-scale datasets
and models in deep learning. While it is computationally beneficial to use
large batch sizes, it often requires a specially designed learning rate (LR)
schedule to achieve a comparable level of performance as in smaller batch
training. Especially, when the number of training epochs is constrained, the
use of a large LR and a warmup strategy is critical in the final performance of
large-batch training due to the reduced number of updating steps. In this work,
we propose an automated LR scheduling algorithm which is effective for neural
network training with a large batch size under the given epoch budget. In
specific, the whole schedule consists of two phases: adaptive warmup and
predefined decay, where the LR is increased until the training loss no longer
decreases and decreased to zero until the end of training. Here, whether the
training loss has reached the minimum value is robustly checked with Gaussian
process smoothing in an online manner with a low computational burden. Coupled
with adaptive stochastic optimizers such as AdamP and LAMB, the proposed
scheduler successfully adjusts the LRs without cumbersome hyperparameter tuning
and achieves comparable or better performances than tuned baselines on various
image classification benchmarks and architectures with a wide range of batch
sizes.

本文提出了一种有效的 LR 调试算法，其中包括自适应的预热和预定义的衰减，通过高斯过程平滑的在线检查方法可以有效地训练具有大批次大小的神经网络。