It has repeatedly been observed that loss minimization by stochastic gradient
descent (SGD) leads to heavy-tailed distributions of neural network parameters.
Here, we analyze a continuous diffusion approximation of SGD, called
homogenized stochastic gradient descent, show that it behaves asymptotically
heavy-tailed, and give explicit upper and lower bounds on its tail-index. We
validate these bounds in numerical experiments and show that they are typically
close approximations to the empirical tail-index of SGD iterates. In addition,
their explicit form enables us to quantify the interplay between optimization
parameters and the tail-index. Doing so, we contribute to the ongoing
discussion on links between heavy tails and the generalization performance of
neural networks as well as the ability of SGD to avoid suboptimal local minima.

通过对连续扩散逼近的随机梯度下降进行分析，我们发现它在渐近意义下表现出重尾分布，并给出了尾指数的上下界。我们通过数值实验验证了这些界限，并显示它们通常是 SGD 迭代的经验尾指数的近似。此外，这些界限的显式形式使我们能够量化优化参数与尾指数之间的相互作用，这对于研究神经网络的广义性能和 SGD 避免次优局部极小值的能力的关联问题具有重要意义。

均值随机梯度下降中的重尾出现

Emergence of heavy tails in homogenized stochastic gradient descent

Diffusion approximation provides weak approximation for stochastic gradient
descent algorithms in a finite time horizon. In this paper, we introduce new
tools motivated by the backward error analysis of numerical stochastic
differential equations into the theoretical framework of diffusion
approximation, extending the validity of the weak approximation from finite to
infinite time horizon. The new techniques developed in this paper enable us to
characterize the asymptotic behavior of constant-step-size SGD algorithms for
strongly convex objective functions, a goal previously unreachable within the
diffusion approximation framework. Our analysis builds upon a truncated formal
power expansion of the solution of a stochastic modified equation arising from
diffusion approximation, where the main technical ingredient is a
uniform-in-time weak error bound controlling the long-term behavior of the
expansion coefficient functions near the global minimum. We expect these new
techniques to greatly expand the range of applicability of diffusion
approximation to cover wider and deeper aspects of stochastic optimization
algorithms in data science.

本文介绍一种新的方法来扩展了扩散逼近的有效性，使得可以利用此方法对强凸目标函数的常步长随机梯度下降算法进行渐进行为的表征，从而使得扩散逼近的适用范围更广、更深入涵盖了数据科学中随机优化算法的应用。

通过扩散逼近对随机梯度下降算法进行时间均匀性弱误差分析

Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent  Algorithms via Diffusion Approximation

In this paper, we propose to adopt the diffusion approximation tools to study
the dynamics of Oja's iteration which is an online stochastic gradient descent
method for the principal component analysis. Oja's iteration maintains a
running estimate of the true principal component from streaming data and enjoys
less temporal and spatial complexities. We show that the Oja's iteration for
the top eigenvector generates a continuous-state discrete-time Markov chain
over the unit sphere. We characterize the Oja's iteration in three phases using
diffusion approximation and weak convergence tools. Our three-phase analysis
further provides a finite-sample error bound for the running estimate, which
matches the minimax information lower bound for principal component analysis
under the additional assumption of bounded samples.

本文提出采用扩散近似工具研究 Oja 迭代的动态特性，它是一种用于数据流中的在线随机梯度下降方法。我们使用扩散近似和弱收敛工具将 Oja 迭代特征为三个阶段，进而为其提供有限样本误差界限。