While stochastic gradient descent (SGD) with momentum has fast convergence and excellent generalizability, a theoretical explanation for this is lacking. In this paper, we show that SGD with momentum smooths the objective function, the degree of which is determined by the learning rate, the batch size, the momentum factor, the variance of the stochastic gradient, and the upper bound of the gradient norm. This theoretical finding reveals why momentum improves generalizability and provides new insights into the role of the hyperparameters, including momentum factor. We also present an implicit graduated optimization algorithm that exploits the smoothing properties of SGD with momentum and provide experimental results supporting our assertion that SGD with momentum smooths the objective function.

随机梯度下降(SGD)与动量在收敛性和泛化能力方面具有快速收敛和优秀的表现，但缺乏理论解释。本文证明了SGD与动量使目标函数平滑化，平滑程度由学习率、批量大小、动量因子、随机梯度的方差和梯度范数的上界决定。这一理论发现揭示了动量为何改善泛化性能，并对包括动量因子在内的超参数的作用提供了新的见解。我们还提出了一种利用SGD与动量平滑性质的隐性渐变优化算法，并提供了支持我们断言的实验结果。

隐式逐步优化中动量在平滑目标函数中的作用