Many modern machine learning applications - from online principal component
analysis to covariance matrix identification and dictionary learning - can be
formulated as minimization problems on Riemannian manifolds, and are typically
solved with a Riemannian stochastic gradient method (or some variant thereof).
However, in many cases of interest, the resulting minimization problem is not
geodesically convex, so the convergence of the chosen solver to a desirable
solution - i.e., a local minimizer - is by no means guaranteed. In this paper,
we study precisely this question, that is, whether stochastic Riemannian
optimization algorithms are guaranteed to avoid saddle points with probability
1. For generality, we study a family of retraction-based methods which, in
addition to having a potentially much lower per-iteration cost relative to
Riemannian gradient descent, include other widely used algorithms, such as
natural policy gradient methods and mirror descent in ordinary convex spaces.
In this general setting, we show that, under mild assumptions for the ambient
manifold and the oracle providing gradient information, the policies under
study avoid strict saddle points / submanifolds with probability 1, from any
initial condition. This result provides an important sanity check for the use
of gradient methods on manifolds as it shows that, almost always, the limit
state of a stochastic Riemannian algorithm can only be a local minimizer.

对于现代机器学习应用中的最小化问题，研究了基于提纯的方法族，证明了在渐进条件下，从任意初始状态出发，研究中的策略几乎总能避免严格鞍点 / 子流形，从而为在流形上使用梯度方法提供了重要的可靠性验证。

黎曼随机优化方法避免严格鞍点

Riemannian stochastic optimization methods avoid strict saddle points

We consider the problem of learning a one-hidden-layer neural network with
non-overlapping convolutional layer and ReLU activation, i.e., $f(\mathbf{Z},
\mathbf{w}, \mathbf{a}) = \sum_j a_j\sigma(\mathbf{w}^T\mathbf{Z}_j)$, in which
both the convolutional weights $\mathbf{w}$ and the output weights $\mathbf{a}$
are parameters to be learned. When the labels are the outputs from a teacher
network of the same architecture with fixed weights $(\mathbf{w}^*,
\mathbf{a}^*)$, we prove that with Gaussian input $\mathbf{Z}$, there is a
spurious local minimizer. Surprisingly, in the presence of the spurious local
minimizer, gradient descent with weight normalization from randomly initialized
weights can still be proven to recover the true parameters with constant
probability, which can be boosted to probability $1$ with multiple restarts. We
also show that with constant probability, the same procedure could also
converge to the spurious local minimum, showing that the local minimum plays a
non-trivial role in the dynamics of gradient descent. Furthermore, a
quantitative analysis shows that the gradient descent dynamics has two phases:
it starts off slow, but converges much faster after several iterations.

研究了在卷积层和 ReLU 激活下的一层神经网络的学习问题，证明了随机初始化并使用归一化权重的梯度下降可以恢复真实参数，但存在虚假局部最小值，且该局部最小值在梯度下降的动力学中起到了重要作用。

梯度下降学习一层卷积神经网络：不必担心虚假局部极小值

Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of  Spurious Local Minima

We show that gradient descent converges to a local minimizer, almost surely
with random initialization. This is proved by applying the Stable Manifold
Theorem from dynamical systems theory.

本文证明了随机初始化的梯度下降会收敛到局部最小值，证明使用了动力系统理论的 Stable Manifold Theorem。