We study the implicit regularization of mini-batch stochastic gradient descent, when applied to the fundamental problem of least squares regression. We leverage a continuous-time stochastic differential equation having the same moments as stochastic gradient descent, which we call stochastic gradient flow. We give a bound on the excess risk of stochastic gradient flow at time $t$, over ridge regression with tuning parameter $\lambda = 1/t$. The bound may be computed from explicit constants (e.g., the mini-batch size, step size, number of iterations), revealing precisely how these quantities drive the excess risk. Numerical examples show the bound can be small, indicating a tight relationship between the two estimators. We give a similar result relating the coefficients of stochastic gradient flow and ridge. These results hold under no conditions on the data matrix $X$, and across the entire optimization path (not just at convergence).

研究隐式正则化的小批量随机梯度下降，以最小二乘回归为基础问题，利用具有与随机梯度下降相同矩的连续时间随机微分方程，称为随机梯度流。给出了随时间t随机梯度流的超额风险的界限，超过了具有调整参数λ=1/t的岭回归，此界限可以从明确的常数（例如小批量大小，步长，迭代次数）计算出来，显示了这些数量如何精确地推动超额风险。数值实验表明，边界可以很小，表明两种估计值之间存在紧密关系。给出了一个类似的结果，将随机梯度流和岭的系数联系起来。这些结果不受数据矩阵X的任何条件限制，并且跨越整个优化路径（不仅仅在收敛处）

随机梯度流对最小二乘隐式正则化