Top-$k$ sparsification has recently been widely used to reduce the
communication volume in distributed deep learning; however, due to Gradient
Accumulation (GA) dilemma, the performance of top-$k$ sparsification is still
limited. Several methods have been proposed to handle the GA dilemma but have
two drawbacks: (1) they are frustrated by the high communication complexity as
they introduce a large amount of extra transmission; (2) they are not flexible
for non-power-of-two numbers of workers. To solve these two problems, we
propose a flexible and efficient sparse communication framework, dubbed SparDL.
SparDL uses the Spar-Reduce-Scatter algorithm to solve the GA dilemma without
additional communication operations and is flexible to any number of workers.
Besides, to further reduce the communication complexity and adjust the
proportion of latency and bandwidth cost in communication complexity, we
propose the Spar-All-Gather algorithm as part of SparDL. Extensive experiments
validate the superiority of SparDL.

提出了 SparDL，一种灵活而高效的稀疏通信框架，使用 Spar-Reduce-Scatter 算法来解决分布式深度学习中的梯度积累问题，并使用 Spar-All-Gather 算法进一步减少通信复杂度并调整延迟和带宽成本的比例，大量实验证明其卓越性能。