Parallel training of neural networks at scale is challenging due to
significant overheads arising from communication. Recently, deep learning
researchers have developed a variety of pruning algorithms that are capable of
pruning (i.e. setting to zero) 80-90% of the parameters in a neural network to
yield sparse subnetworks that equal the accuracy of the unpruned parent
network. In this work, we propose a novel approach that exploits these sparse
subnetworks to optimize the memory utilization and communication in two popular
algorithms for parallel deep learning namely -- data and inter-layer
parallelism. We integrate our approach into AxoNN, a highly scalable framework
for parallel deep learning that relies on data and inter-layer parallelism, and
demonstrate the reduction in communication time and memory utilization. On 512
NVIDIA V100 GPUs, our optimizations reduce the memory consumption of a 2.7
billion parameter model by 74%, and the total communication time by 40%, thus
providing an overall speedup of 34% over AxoNN, 32% over DeepSpeed-3D and 46%
over Sputnik, a sparse matrix computation baseline.

本文提出利用稀疏子网络以优化内存利用和通信的方法来加速深度学习中的数据和层间并行算法，并将其融入 AxoNN，实验证明在 512 NVIDIA V100 GPU 上，相比于 AxoNN，DeepSpeed-3D 和 Sputnik，优化后的模型内存消耗减少了 74％，总通信时间减少了 40％，从而提供了 34％的总体加速。

利用稀疏性在剪枝神经网络中优化大模型训练

Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training

This paper studies the problem of error-runtime trade-off, typically
encountered in decentralized training based on stochastic gradient descent
(SGD) using a given network. While a denser (sparser) network topology results
in faster (slower) error convergence in terms of iterations, it incurs more
(less) communication time/delay per iteration. In this paper, we propose
MATCHA, an algorithm that can achieve a win-win in this error-runtime trade-off
for any arbitrary network topology. The main idea of MATCHA is to parallelize
inter-node communication by decomposing the topology into matchings. To
preserve fast error convergence speed, it identifies and communicates more
frequently over critical links, and saves communication time by using other
links less frequently. Experiments on a suite of datasets and deep neural
networks validate the theoretical analyses and demonstrate that MATCHA takes up
to $5\times$ less time than vanilla decentralized SGD to reach the same
training loss.

本文研究了分布式训练中通常遇到的误差 - 运行时权衡问题，提出了 MATCHA 算法，该算法能够在任意网络拓扑结构下实现误差 - 运行时权衡的双赢，并且通过将拓扑结构分解为匹配来实现节点之间的并行交流。通过实验验证，MATCHA 算法在达到相同的训练损失时比基本的分布式随机梯度下降算法少花费高达 5 倍的时间。

MATCHA: 基于匹配分解采样的分布式随机梯度下降加速算法

MATCHA: Speeding Up Decentralized SGD via Matching Decomposition  Sampling

High network communication cost for synchronizing gradients and parameters is
the well-known bottleneck of distributed training. In this work, we propose
TernGrad that uses ternary gradients to accelerate distributed deep learning in
data parallelism. Our approach requires only three numerical levels {-1,0,1},
which can aggressively reduce the communication time. We mathematically prove
the convergence of TernGrad under the assumption of a bound on gradients.
Guided by the bound, we propose layer-wise ternarizing and gradient clipping to
improve its convergence. Our experiments show that applying TernGrad on AlexNet
does not incur any accuracy loss and can even improve accuracy. The accuracy
loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a
performance model is proposed to study the scalability of TernGrad. Experiments
show significant speed gains for various deep neural networks. Our source code
is available.

本文提出 TernGrad，使用三元梯度来加速数据并行的分布式深度学习，结果表明可在不损失准确率的前提下缩减通信时间并提高收敛性能，使得多种深度神经网络的速度显著提升。