For infinitesimal learning rates, stochastic gradient descent (SGD) follows
the path of gradient flow on the full batch loss function. However moderately
large learning rates can achieve higher test accuracies, and this
generalization benefit is not explained by convergence bounds, since the
learning rate which maximizes test accuracy is often larger than the learning
rate which minimizes training loss. To interpret this phenomenon we prove that
for SGD with random shuffling, the mean SGD iterate also stays close to the
path of gradient flow if the learning rate is small and finite, but on a
modified loss. This modified loss is composed of the original loss function and
an implicit regularizer, which penalizes the norms of the minibatch gradients.
Under mild assumptions, when the batch size is small the scale of the implicit
regularization term is proportional to the ratio of the learning rate to the
batch size. We verify empirically that explicitly including the implicit
regularizer in the loss can enhance the test accuracy when the learning rate is
small.

本文研究随机梯度下降（SGD）的学习率对准确性的影响，证明当学习率适当大时，SGD 的迭代路径离梯度下降路径更近，这种现象可通过引入一个隐式正则化项进行解释，并通过实验证明在适当的学习率下包含隐式正则化项可以提高测试准确性。