One of the major issues in stochastic gradient descent (SGD) methods is how
to choose an appropriate step size while running the algorithm. Since the
traditional line search technique does not apply for stochastic optimization
algorithms, the common practice in SGD is either to use a diminishing step
size, or to tune a fixed step size by hand, which can be time consuming in
practice. In this paper, we propose to use the Barzilai-Borwein (BB) method to
automatically compute step sizes for SGD and its variant: stochastic variance
reduced gradient (SVRG) method, which leads to two algorithms: SGD-BB and
SVRG-BB. We prove that SVRG-BB converges linearly for strongly convex objective
functions. As a by-product, we prove the linear convergence result of SVRG with
Option I proposed in [10], whose convergence result is missing in the
literature. Numerical experiments on standard data sets show that the
performance of SGD-BB and SVRG-BB is comparable to and sometimes even better
than SGD and SVRG with best-tuned step sizes, and is superior to some advanced
SGD variants.

该研究使用 Barzilai-Borwein 方法自动计算 Stochastic Gradient Descent（SGD）和 Stochastic Variance Reduced Gradient（SVRG）算法的步长，证明了 SVRG-BB 在强凸目标函数上线性收敛的结果，并在标准数据集上进行了数字实验，得出了 SGD-BB 和 SVRG-BB 相对于 SGD 和 SVRG 在性能上更好的结论。