We give quantitative estimates for the rate of convergence of Riemannian
stochastic gradient descent (RSGD) to Riemannian gradient flow and to a
diffusion process, the so-called Riemannian stochastic modified flow (RSMF).
Using tools from stochastic differential geometry we show that, in the small
learning rate regime, RSGD can be approximated by the solution to the RSMF
driven by an infinite-dimensional Wiener process. The RSMF accounts for the
random fluctuations of RSGD and, thereby, increases the order of approximation
compared to the deterministic Riemannian gradient flow. The RSGD is build using
the concept of a retraction map, that is, a cost efficient approximation of the
exponential map, and we prove quantitative bounds for the weak error of the
diffusion approximation under assumptions on the retraction map, the geometry
of the manifold, and the random estimators of the gradient.

给出了 Riemannian 随机梯度下降（RSGD）到 Riemannian 梯度流和所谓的 Riemannian 随机修改流（RSMF）的收敛速率的定量估计。使用了随机微分几何的工具，我们展示了在小学习率情况下，RSGD 可以通过由无穷维威纳过程驱动的 RSMF 的解来近似。RSMF 考虑了 RSGD 的随机波动，并且相对于确定性的 Riemannian 梯度流提高了近似的阶数。RSGD 使用了一个投射映射的概念，即，一个对指数映射的成本有效逼近，并且在对投射映射、流形的几何和梯度的随机估计作出假设的情况下，我们证明了扩散逼近的弱误差的定量界限。

随机修改流在黎曼随机梯度下降中的应用

Stochastic Modified Flows for Riemannian Stochastic Gradient Descent

We study the convergence of gradient flows related to learning deep linear
neural networks (where the activation function is the identity map) from data.
In this case, the composition of the network layers amounts to simply
multiplying the weight matrices of all layers together, resulting in an
overparameterized problem. The gradient flow with respect to these factors can
be re-interpreted as a Riemannian gradient flow on the manifold of rank-$r$
matrices endowed with a suitable Riemannian metric. We show that the flow
always converges to a critical point of the underlying functional. Moreover, we
establish that, for almost all initializations, the flow converges to a global
minimum on the manifold of rank $k$ matrices for some $k\leq r$.

本文研究了与从数据中学习深度线性神经网络（其中激活函数为恒等映射）相关的梯度流的收敛性，结果表明梯度流总是收敛于潜在函数的临界点。