Learning Rate (LR) is an important hyper-parameter to tune for effective
training of deep neural networks (DNNs). Even for the baseline of a constant
learning rate, it is non-trivial to choose a good constant value for training a
DNN. Dynamic learning rates involve multi-step tuning of LR values at various
stages of the training process and offer high accuracy and fast convergence.
However, they are much harder to tune. In this paper, we present a
comprehensive study of 13 learning rate functions and their associated LR
policies by examining their range parameters, step parameters, and value update
parameters. We propose a set of metrics for evaluating and selecting LR
policies, including the classification confidence, variance, cost, and
robustness, and implement them in LRBench, an LR benchmarking system. LRBench
can assist end-users and DNN developers to select good LR policies and avoid
bad LR policies for training their DNNs. We tested LRBench on Caffe, an open
source deep learning framework, to showcase the tuning optimization of LR
policies. Evaluated through extensive experiments, we attempt to demystify the
tuning of LR policies by identifying good LR policies with effective LR value
ranges and step sizes for LR update schedules.

本文基于 LRBench 系统对 13 种学习率函数以及相应的学习率策略进行研究，提出了一组评估和选择学习率策略的度量标准，包括分类置信度、方差、成本和鲁棒性，表明 LRBench 辅助用户选择好的学习率策略，避免训练深度神经网络时出现不良策略。