Background: Distributed training is essential for large scale training of
deep neural networks (DNNs). The dominant methods for large scale DNN training
are synchronous (e.g. All-Reduce), but these require waiting for all workers in
each step. Thus, these methods are limited by the delays caused by straggling
workers. Results: We study a typical scenario in which workers are straggling
due to variability in compute time. We find an analytical relation between
compute time properties and scalability limitations, caused by such straggling
workers. With these findings, we propose a simple yet effective decentralized
method to reduce the variation among workers and thus improve the robustness of
synchronous training. This method can be integrated with the widely used
All-Reduce. Our findings are validated on large-scale training tasks using 200
Gaudi Accelerators.

本文提出了一种简单而有效的分散式方法，以减少工作人员之间的变化，从而提高同步训练的鲁棒性，并验证了这些结果。

DropCompute：通过计算方差减少计算量的简单而稳健的分布式同步训练

DropCompute: simple and more robust distributed synchronous training via  compute variance reduction

Gradient descent (GD) methods are commonly employed in machine learning
problems to optimize the parameters of the model in an iterative fashion. For
problems with massive datasets, computations are distributed to many parallel
computing servers (i.e., workers) to speed up GD iterations. While distributed
computing can increase the computation speed significantly, the per-iteration
completion time is limited by the slowest straggling workers. Coded distributed
computing can mitigate straggling workers by introducing redundant
computations; however, existing coded computing schemes are mainly designed
against persistent stragglers, and partial computations at straggling workers
are discarded, leading to wasted computational capacity. In this paper, we
propose a novel gradient coding (GC) scheme which allows multiple coded
computations to be conveyed from each worker to the master per iteration. We
numerically show that the proposed GC with multi-message communication (MMC)
together with clustering provides significant improvements in the average
completion time (of each iteration), with minimal or no increase in the
communication load.

本文介绍了一种新的梯度编码（GC）方案，结合多消息通信（MMC）和聚类，以减少因服务器慢导致的迭代时间延长问题，其能有效提高平均完成时间并降低通信负荷。