Training the deep convolutional neural network for computer vision problems
is slow and inefficient, especially when it is large and distributed across
multiple devices. The inefficiency is caused by the backpropagation algorithm's
forward locking, backward locking, and update locking problems. Existing
solutions for acceleration either can only handle one locking problem or lead
to severe accuracy loss or memory inefficiency. Moreover, none of them consider
the straggler problem among devices. In this paper, we propose Layer-wise
Staleness and a novel efficient training algorithm, Diversely Stale Parameters
(DSP), to address these challenges. We also analyze the convergence of DSP with
two popular gradient-based methods and prove that both of them are guaranteed
to converge to critical points for non-convex problems. Finally, extensive
experimental results on training deep learning models demonstrate that our
proposed DSP algorithm can achieve significant training speedup with stronger
robustness than compared methods.

本研究提出了 Layer-wise Staleness 方法与 Diversely Stale Parameters (DSP) 算法来解决深度卷积神经网络的训练效率问题，并对两种梯度法进行分析和证明，实验证明 DSP 能显著提升训练速度并具备更强的鲁棒性。