Distributed model training is vulnerable to byzantine system failures and
adversarial compute nodes, i.e., nodes that use malicious updates to corrupt
the global model stored at a parameter server (PS). To guarantee some form of
robustness, recent work suggests using variants of the geometric median as an
aggregation rule, in place of gradient averaging. Unfortunately, median-based
rules can incur a prohibitive computational overhead in large-scale settings,
and their convergence guarantees often require strong assumptions. In this
work, we present DRACO, a scalable framework for robust distributed training
that uses ideas from coding theory. In DRACO, each compute node evaluates
redundant gradients that are used by the parameter server to eliminate the
effects of adversarial updates. DRACO comes with problem-independent robustness
guarantees, and the model that it trains is identical to the one trained in the
adversary-free setup. We provide extensive experiments on real datasets and
distributed setups across a variety of large-scale models, where we show that
DRACO is several times, to orders of magnitude faster than median-based
approaches.

通过编码理论，我们提出了 DRACO，一个可伸缩的分布式训练框架，用于解决分布式模型训练中的拜占庭系统故障和对抗计算节点等问题，DRACO 具有问题无关的鲁棒性保证，并且训练的模型与无对手的设置中训练的模型相同。