We propose a statistical adaptive procedure called SALSA for automatically
scheduling the learning rate (step size) in stochastic gradient methods. SALSA
first uses a smoothed stochastic line-search procedure to gradually increase
the learning rate, then automatically switches to a statistical method to
decrease the learning rate. The line search procedure ``warms up'' the
optimization process, reducing the need for expensive trial and error in
setting an initial learning rate. The method for decreasing the learning rate
is based on a new statistical test for detecting stationarity when using a
constant step size. Unlike in prior work, our test applies to a broad class of
stochastic gradient algorithms without modification. The combined method is
highly robust and autonomous, and it matches the performance of the best
hand-tuned learning rate schedules in our experiments on several deep learning
tasks.

提出一种名为 SALSA 的统计自适应程序，用于自动调整随机梯度方法中的学习率（步长），该方法使用平滑的随机线性搜索程序逐渐增加学习率，然后自动转换为一个新的统计方法来降低学习率，它在广泛的随机梯度算法中使用了一个新的统计测试来检测状态，能够适应多类深度学习任务。