We study adversary-resilient stochastic distributed optimization, in which
$m$ machines can independently compute stochastic gradients, and cooperate to
jointly optimize over their local objective functions. However, an
$\alpha$-fraction of the machines are $\textit{Byzantine}$, in that they may
behave in arbitrary, adversarial ways. We consider a variant of this procedure
in the challenging $\textit{non-convex}$ case. Our main result is a new
algorithm SafeguardSGD which can provably escape saddle points and find
approximate local minima of the non-convex objective. The algorithm is based on
a new concentration filtering technique, and its sample and time complexity
bounds match the best known theoretical bounds in the stochastic, distributed
setting when no Byzantine machines are present.
Our algorithm is very practical: it improves upon the performance of all
prior methods when training deep neural networks, it is relatively lightweight,
and it is the first method to withstand two recently-proposed Byzantine
attacks.

本文研究对抗鲁棒的分布式随机优化，并提出了一种新算法 SafeguardSGD，它可以确定性地避开鞍点并找到非凸目标的近似局部最小值。该算法基于一种新的集中过滤技术，可证明其样本和时间复杂性边界与之前的最佳理论边界相匹配，同时它也是第一种能够抵御两种最近提出的拜占庭攻击的方法。该算法在训练深度神经网络方面的表现优于所有先前的方法。

拜占庭容错的非凸随机梯度下降

Byzantine-Resilient Non-Convex Stochastic Gradient Descent

Understanding of GAN training is still very limited. One major challenge is
its non-convex-non-concave min-max objective, which may lead to sub-optimal
local minima. In this work, we perform a global landscape analysis of the
empirical loss of GANs. We prove that a class of separable-GAN, including the
original JS-GAN, has exponentially many bad basins which are perceived as
mode-collapse. We also study the relativistic pairing GAN (RpGAN) loss which
couples the generated samples and the true samples. We prove that RpGAN has no
bad basins. Experiments on synthetic data show that the predicted bad basin can
indeed appear in training. We also perform experiments to support our theory
that RpGAN has a better landscape than separable-GAN. For instance, we
empirically show that RpGAN performs better than separable-GAN with relatively
narrow neural nets. The code is available at this https URL

本文通过对生成对抗网络的全局概貌的分析，证明了一类可分离生成对抗网络存在大量的坏基底，相较之下，使用 RpGAN 合相对更好。