In this paper, we propose a class of robust stochastic subgradient methods for distributed learning from heterogeneous datasets at presence of an unknown number of Byzantine workers. The Byzantine workers, during the learning process, may send arbitrary incorrect messages to the master due to data corruptions, communication failures or malicious attacks, and consequently bias the learned model. The key to the proposed methods is a regularization term incorporated with the objective function so as to robustify the learning task and mitigate the negative effects of Byzantine attacks. The resultant subgradient-based algorithms are termed Byzantine-Robust Stochastic Aggregation methods, justifying our acronym RSA used henceforth. In contrast to most of the existing algorithms, RSA does not rely on the assumption that the data are independent and identically distributed (i.i.d.) on the workers, and hence fits for a wider class of applications. Theoretically, we show that: i) RSA converges to a near-optimal solution with the learning error dependent on the number of Byzantine workers; ii) the convergence rate of RSA under Byzantine attacks is the same as that of the stochastic gradient descent method, which is free of Byzantine attacks. Numerically, experiments on real dataset corroborate the competitive performance of RSA and a complexity reduction compared to the state-of-the-art alternatives.

本文提出了一种适用于异构数据集的分布式学习的鲁棒性随机次梯度方法——Byzantine-Robust Stochastic Aggregation (RSA)方法，用来应对Byzantine工人的攻击。RSA方法不依赖于数据是独立同分布的假设，具有更广泛的适用性。理论上，RSA收敛到一个接近最优解的解决方案，学习错误取决于Byzantine工作人员数目，收敛速率与没有Byzantine工人的随机梯度下降方法相同。数值实验表明，RSA具有竞争性的性能和与现有算法相比较的复杂度降低。

RSA: 分布式异构数据学习的拜占庭容错随机聚合方法