We study adversary-resilient stochastic distributed optimization, in which
$m$ machines can independently compute stochastic gradients, and cooperate to
jointly optimize over their local objective functions. However, an
$\alpha$-fraction of the machines are $\textit{Byzantine}$, in that they may
behave in arbitrary, adversarial ways. We consider a variant of this procedure
in the challenging $\textit{non-convex}$ case. Our main result is a new
algorithm SafeguardSGD which can provably escape saddle points and find
approximate local minima of the non-convex objective. The algorithm is based on
a new concentration filtering technique, and its sample and time complexity
bounds match the best known theoretical bounds in the stochastic, distributed
setting when no Byzantine machines are present.
Our algorithm is very practical: it improves upon the performance of all
prior methods when training deep neural networks, it is relatively lightweight,
and it is the first method to withstand two recently-proposed Byzantine
attacks.

本文研究对抗鲁棒的分布式随机优化，并提出了一种新算法 SafeguardSGD，它可以确定性地避开鞍点并找到非凸目标的近似局部最小值。该算法基于一种新的集中过滤技术，可证明其样本和时间复杂性边界与之前的最佳理论边界相匹配，同时它也是第一种能够抵御两种最近提出的拜占庭攻击的方法。该算法在训练深度神经网络方面的表现优于所有先前的方法。