Many emerging AI applications request distributed machine learning (ML) among
edge systems (e.g., IoT devices and PCs at the edge of the Internet), where
data cannot be uploaded to a central venue for model training, due to their
large volumes and/or security/privacy concerns. Edge devices are intrinsically
heterogeneous in computing capacity, posing significant challenges to parameter
synchronization for parallel training with the parameter server (PS)
architecture. This paper proposes ADSP, a parameter synchronization scheme for
distributed machine learning (ML) with heterogeneous edge systems. Eliminating
the significant waiting time occurring with existing parameter synchronization
models, the core idea of ADSP is to let faster edge devices continue training,
while committing their model updates at strategically decided intervals. We
design algorithms that decide time points for each worker to commit its model
update, and ensure not only global model convergence but also faster
convergence. Our testbed implementation and experiments show that ADSP
outperforms existing parameter synchronization models significantly in terms of
ML model convergence time, scalability and adaptability to large heterogeneity.

提出 ADSP 参数同步方案，解决了分布式机器学习中参数同步与边缘设备的异构性问题，通过策略性地决定每个工作节点提交模型更新的时间点，不仅确保了全局模型收敛，而且加快了收敛速度。

分布式机器学习基于异构边缘系统

Distributed Machine Learning through Heterogeneous Edge Systems

Model parameter synchronization across GPUs introduces high overheads for
data-parallel training at scale. Existing parameter synchronization protocols
cannot effectively leverage available network resources in the face of ever
increasing hardware heterogeneity. To address this, we propose Blink, a
collective communication library that dynamically generates optimal
communication primitives by packing spanning trees. We propose techniques to
minimize the number of trees generated and extend Blink to leverage
heterogeneous communication channels for faster data transfers. Evaluations
show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8x
faster model synchronization, and reduce end-to-end training time for image
classification tasks by up to 40%.

Blink 是一种集合通信库，它通过打包生成生成最佳通讯原语的生成树来解决 GPU 上的参数同步问题，而且还可以利用异构通信通道实现更快的数据传输，从而优化数据并行训练，提高图像分类任务的效率。

Blink: 分布式机器学习的快速通用集合

Blink: Fast and Generic Collectives for Distributed ML

Data parallel training is widely used for scaling distributed deep neural
network (DNN) training. However, the performance benefits are often limited by
the communication-heavy parameter synchronization step. In this paper, we take
advantage of the domain specific knowledge of DNN training and overlap
parameter synchronization with computation in order to improve the training
performance. We make two key observations: (1) the optimal data representation
granularity for the communication may differ from that used by the underlying
DNN model implementation and (2) different parameters can afford different
synchronization delays. Based on these observations, we propose a new
synchronization mechanism called Priority-based Parameter Propagation (P3). P3
synchronizes parameters at a finer granularity and schedules data transmission
in such a way that the training process incurs minimal communication delay. We
show that P3 can improve the training throughput of ResNet-50, Sockeye and
VGG-19 by as much as 25%, 38% and 66% respectively on clusters with realistic
network bandwidth

本文提出了一种名为 “P3” 的新型同步机制，通过优化参数同步和计算之间的交替，以最小化通信延迟，从而提高数据并行训练的性能，并在 ResNet-50，Sockeye 和 VGG-19 等模型上实现了高达 25％，38％和 66％的训练吞吐量提升。