With the fast development of deep learning, people have started to train very big neural networks using massive data. Asynchronous Stochastic Gradient Descent (ASGD) is widely used to fulfill this task, which, however, is known to suffer from the problem of delayed gradient. That is, when a local worker adds the gradient it calculates to the global model, the global model may have been updated by other workers and this gradient becomes "delayed". We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD. This is done by leveraging Taylor expansion of the gradient function and efficient approximators to the Hessian matrix of the loss function. We call the corresponding new algorithm Delay Compensated ASGD (DC-ASGD). We evaluated the proposed algorithm on CIFAR-10 and ImageNet datasets, and experimental results demonstrate that DC-ASGD can outperform both synchronous SGD and ASGD, and nearly approaches the performance of sequential SGD.

提出了一种新的异步随机梯度下降算法，通过利用梯度函数的泰勒展开和损失函数的快速近似海森矩阵，实现了延迟补偿，从而使ASGD的优化行为更接近于顺序SGD。在CIFAR-10和ImageNet数据集上进行了评估，实验结果表明，DC-ASGD优于同步SGD和异步SGD，几乎接近于顺序SGD的性能。

具有延迟补偿的异步随机梯度下降