Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates. This variance also changes over the optimization process; when using a constant batch size, stability and convergence is thus often enforced by means of a (manually tuned) decreasing learning rate schedule. We propose a practical method for dynamic batch size adaptation. It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance proportionally to the value of the objective function, removing the need for the aforementioned learning rate decrease. In contrast to recent related work, our algorithm couples the batch size to the learning rate, directly reflecting the known relationship between the two. On three image classification benchmarks, our batch size adaptation yields faster optimization convergence, while simultaneously simplifying learning rate tuning. A TensorFlow implementation is available.

本论文提出了动态批次大小自适应方法，通过估计随机梯度的方差并将批次大小调整为与目标函数成比例的方式，从而消除了先前所述的学习速率降低的需要。与相关工作不同的是，我们的算法将批次大小与学习速率相结合，直接反映了两者之间的已知关系。在流行的图像分类基准测试中，我们的批次大小自适应可以更快地优化收敛，同时简化学习速率调整。

将自适应批量大小与学习率相结合