Distributed training is the de facto standard to scale up the training of
Deep Neural Networks (DNNs) with multiple GPUs. The performance bottleneck of
distributed training lies in communications for gradient synchronization.
Recently, practitioners have observed sparsity in gradient tensors, suggesting
the potential to reduce the traffic volume in communication and improve
end-to-end training efficiency. Yet, the optimal communication scheme to fully
leverage sparsity is still missing. This paper aims to address this gap. We
first analyze the characteristics of sparse tensors in popular DNN models to
understand the fundamentals of sparsity. We then systematically explore the
design space of communication schemes for sparse tensors and find the optimal
one. % We then find the optimal scheme based on the characteristics by
systematically exploring the design space. We also develop a gradient
synchronization system called Zen that approximately realizes it for sparse
tensors. We demonstrate that Zen can achieve up to 5.09x speedup in
communication time and up to 2.48x speedup in training throughput compared to
the state-of-the-art methods.

这篇论文研究了在分布式训练深度神经网络中，梯度同步的通信性能瓶颈问题，并发现了梯度张量的稀疏性可以减少通信量从而提高整体训练效率，通过系统地探索通信方案的设计空间，找到了最佳方案，并开发了名为 Zen 的梯度同步系统，通过对稀疏张量的近似实现可以实现高达 5.09 倍的通信时间加速和 2.48 倍的训练吞吐量加速。

Zen: 高效稀疏张量同步的分布式深度神经网络训练

Zen: Near-Optimal Sparse Tensor Synchronization for Distributed DNN  Training

Mega-constellations of small-size Low Earth Orbit (LEO) satellites are
currently planned and deployed by various private and public entities. While
global connectivity is the main rationale, these constellations also offer the
potential to gather immense amounts of data, e.g., for Earth observation. Power
and bandwidth constraints together with motives like privacy, limiting delay,
or resiliency make it desirable to process this data directly within the
constellation. We consider the implementation of on-board federated learning
(FL) orchestrated by an out-of-constellation parameter server (PS) and propose
a novel communication scheme tailored to support FL. It leverages intra-orbit
inter-satellite links, the predictability of satellite movements and partial
aggregating to massively reduce the training time and communication costs. In
particular, for a constellation with 40 satellites equally distributed among
five low Earth orbits and the PS in medium Earth orbit, we observe a 29x
speed-up in the training process time and a 8x traffic reduction at the PS over
the baseline.

本文提出了一种基于联邦学习的星座内数据处理方案，通过利用轨道间通信、卫星轨道预测以及部分聚合等技术，在星座内实现海量数据的高效训练和通信传输。在此方案的基础上，对于 40 颗卫星分布在 5 个低地球轨道上，以及参数服务器在中地球轨道上的星座，相比于基线，训练时间加速 29 倍，参数服务器上的通信传输量减少了 8 倍。