Asynchronous momentum stochastic gradient descent algorithms (Async-MSGD) is
one of the most popular algorithms in distributed machine learning. However,
its convergence properties for these complicated nonconvex problems is still
largely unknown, because of the current technical limit. Therefore, in this
paper, we propose to analyze the algorithm through a simpler but nontrivial
nonconvex problem - streaming PCA, which helps us to understand Aync-MSGD
better even for more general problems. Specifically, we establish the
asymptotic rate of convergence of Async-MSGD for streaming PCA by diffusion
approximation. Our results indicate a fundamental tradeoff between asynchrony
and momentum: To ensure convergence and acceleration through asynchrony, we
have to reduce the momentum (compared with Sync-MSGD). To the best of our
knowledge, this is the first theoretical attempt on understanding Async-MSGD
for distributed nonconvex stochastic optimization. Numerical experiments on
both streaming PCA and training deep neural networks are provided to support
our findings for Async-MSGD.

本文通过对简单但非平凡的非凸问题 - 流式 PCA 的分析，建立了异步动量随机梯度下降算法 (Async-MSGD) 在流式 PCA 中渐进收敛速率的扩散近似，结果表明异步和动量之间存在根本的权衡：为了通过异步获取收敛和加速，我们必须降低动量 (与 Sync-MSGD 相比)，这是对异步 - MSGD 在分布式非凸随机优化中理解的第一次理论尝试。