Distributed learning has emerged as a leading paradigm for training large
machine learning models. However, in real-world scenarios, participants may be
unreliable or malicious, posing a significant challenge to the integrity and
accuracy of the trained models. Byzantine fault tolerance mechanisms have been
proposed to address these issues, but they often assume full participation from
all clients, which is not always practical due to the unavailability of some
clients or communication constraints. In our work, we propose the first
distributed method with client sampling and provable tolerance to Byzantine
workers. The key idea behind the developed method is the use of gradient
clipping to control stochastic gradient differences in recursive variance
reduction. This allows us to bound the potential harm caused by Byzantine
workers, even during iterations when all sampled clients are Byzantine.
Furthermore, we incorporate communication compression into the method to
enhance communication efficiency. Under quite general assumptions, we prove
convergence rates for the proposed method that match the existing
state-of-the-art (SOTA) theoretical results.

提出了一种具有客户端抽样和对拜占庭工作者的可证明容错性的分布式方法。通过梯度剪裁控制递归方差减少中的随机梯度差异来限制拜占庭工作者可能造成的潜在危害，并且结合通信压缩来提高通信效率。在相当一般的假设下，证明了该方法的收敛速度与现有的理论结果相匹配。

拜占庭健壮性和部分参与可以同时实现：只需裁剪梯度差异

Byzantine Robustness and Partial Participation Can Be Achieved  Simultaneously: Just Clip Gradient Differences

This paper proposes a Robust Gradient Classification Framework (RGCF) for
Byzantine fault tolerance in distributed stochastic gradient descent. The
framework consists of a pattern recognition filter which we train to be able to
classify individual gradients as Byzantine by using their direction alone. This
filter is robust to an arbitrary number of Byzantine workers for convex as well
as non-convex optimisation settings, which is a significant improvement on the
prior work that is robust to Byzantine faults only when up to 50% of the
workers are Byzantine. This solution does not require an estimate of the number
of Byzantine workers; its running time is not dependent on the number of
workers and can scale up to training instances with a large number of workers
without a loss in performance. We validate our solution by training
convolutional neural networks on the MNIST dataset in the presence of Byzantine
workers.

本文提出了一个 Robust Gradient Classification Framework（RGCF）用于在分布式随机梯度下降中实现拜占庭容错。该框架包括一个模式识别过滤器，我们通过仅使用梯度的方向训练它，使其能够将单个梯度分类为拜占庭式的。此解决方案不需要估计拜占庭式工作者的数量；其运行时间不依赖于工作者的数量，并且可以扩展到具有大量工作者的训练实例而不会丧失性能。