In this work, we consider the resilience of distributed algorithms based on stochastic gradient descent (SGD) in distributed learning with potentially Byzantine attackers, who could send arbitrary information to the parameter server to disrupt the training process. Toward this end, we propose a new Lipschitz-inspired coordinate-wise median approach (LICM-SGD) to mitigate Byzantine attacks. We show that our LICM-SGD algorithm can resist up to half of the workers being Byzantine attackers, while still converging almost surely to a stationary region in non-convex settings. Also, our LICM-SGD method does not require any information about the number of attackers and the Lipschitz constant, which makes it attractive for practical implementations. Moreover, our LICM-SGD method enjoys the optimal $O(md)$ computational time-complexity in the sense that the time-complexity is the same as that of the standard SGD under no attacks. We conduct extensive experiments to show that our LICM-SGD algorithm consistently outperforms existing methods in training multi-class logistic regression and convolutional neural networks with MNIST and CIFAR-10 datasets. In our experiments, LICM-SGD also achieves a much faster running time thanks to its low computational time-complexity.

本文提出了一种新的利普希茨坐标中值方法(LICM-SGD)来降低拜占庭攻击对分布式学习中基于随机梯度下降(SGD)的分布式算法的影响。该算法不需要关于攻击者数量和利普希茨常数的任何信息，在实现中具有吸引力，并且在多分类逻辑回归和卷积神经网络的训练中的MNIST和CIFAR-10数据集上，该算法始终表现出色，并且具有更快的运行时间。

分布式学习中的拜占庭鲁棒随机梯度下降：一种基于Lipschitz的逐坐标中位数方法