Robust distributed learning consists in achieving good learning performance
despite the presence of misbehaving workers. State-of-the-art (SOTA) robust
distributed gradient descent (Robust-DGD) methods, relying on robust
aggregation, have been proven to be optimal: Their learning error matches the
lower bound established under the standard heterogeneity model of $(G,
B)$-gradient dissimilarity. The learning guarantee of SOTA Robust-DGD cannot be
further improved when model initialization is done arbitrarily. However, we
show that it is possible to circumvent the lower bound, and improve the
learning performance, when the workers' gradients at model initialization are
assumed to be bounded. We prove this by proposing pre-aggregation clipping of
workers' gradients, using a novel scheme called adaptive robust clipping (ARC).
Incorporating ARC in Robust-DGD provably improves the learning, under the
aforementioned assumption on model initialization. The factor of improvement is
prominent when the tolerable fraction of misbehaving workers approaches the
breakdown point. ARC induces this improvement by constricting the search space,
while preserving the robustness property of the original aggregation scheme at
the same time. We validate this theoretical finding through exhaustive
experiments on benchmark image classification tasks.

通过引入新的方案，即自适应鲁棒剪裁 (ARC)，在模型初始化时限制工作人员梯度，从而改善鲁棒分布式梯度下降 (Robust-DGD) 的学习性能，特别是在可容忍的错误工作人员比例接近失效点时，该改进因子更为突出。

通过裁剪梯度提升分布式学习的鲁棒性

Boosting Robustness by Clipping Gradients in Distributed Learning

Robust distributed learning with Byzantine failures has attracted extensive
research interests in recent years. However, most of existing methods suffer
from curse of dimensionality, which is increasingly serious with the growing
complexity of modern machine learning models. In this paper, we design a new
method that is suitable for high dimensional problems, under arbitrary number
of Byzantine attackers. The core of our design is a direct high dimensional
semi-verified mean estimation method. Our idea is to identify a subspace first.
The components of mean value perpendicular to this subspace can be estimated
via gradient vectors uploaded from worker machines, while the components within
this subspace are estimated using auxiliary dataset. We then use our new method
as the aggregator of distributed learning problems. Our theoretical analysis
shows that the new method has minimax optimal statistical rates. In particular,
the dependence on dimensionality is significantly improved compared with
previous works.

我们设计了一个适用于高维问题的新方法，可以在任意数量的拜占庭攻击者下进行高效分布式学习，并采用半验证的均值估计方法实现，通过从工作机上传的梯度向量估计均值值的分量以及通过辅助数据集估计子空间内的分量，最终以我们的新方法作为分布式学习聚合器。理论分析表明，该新方法具有极小极大统计速率，并且与之前的方法相比，在维度依赖性方面得到了显着改进。