With the rapid growth in the volume of data sets, models, and devices in the
domain of deep learning, there is increasing attention on large-scale
distributed deep learning. In contrast to traditional distributed deep
learning, the large-scale scenario poses new challenges that include fault
tolerance, scalability of algorithms and infrastructures, and heterogeneity in
data sets, models, and resources. Due to intensive synchronization of models
and sharing of data across GPUs and computing nodes during distributed training
and inference processes, communication efficiency becomes the bottleneck for
achieving high performance at a large scale. This article surveys the
literature over the period of 2018-2023 on algorithms and technologies aimed at
achieving efficient communication in large-scale distributed deep learning at
various levels, including algorithms, frameworks, and infrastructures.
Specifically, we first introduce efficient algorithms for model synchronization
and communication data compression in the context of large-scale distributed
training. Next, we introduce efficient strategies related to resource
allocation and task scheduling for use in distributed training and inference.
After that, we present the latest technologies pertaining to modern
communication infrastructures used in distributed deep learning with a focus on
examining the impact of the communication overhead in a large-scale and
heterogeneous setting. Finally, we conduct a case study on the distributed
training of large language models at a large scale to illustrate how to apply
these technologies in real cases. This article aims to offer researchers a
comprehensive understanding of the current landscape of large-scale distributed
deep learning and to reveal promising future research directions toward
communication-efficient solutions in this scope.

这篇文章调查了 2018 年至 2023 年期间关于算法和技术的文献，旨在实现大规模分布式深度学习中的高效通信，包括算法、框架和基础设施。具体而言，我们首先介绍了在大规模分布式训练中模型同步和通信数据压缩方面的高效算法。接下来，我们介绍了与分布式训练和推理中资源分配和任务调度相关的高效策略。此后，我们介绍了现代通信基础设施中使用的最新技术，重点研究了大规模和异构环境下通信开销的影响。最后，我们通过对大规模语言模型的分布式训练进行案例研究，展示了如何在实际情况中应用这些技术。这篇文章旨在为研究人员提供对大规模分布式深度学习的现状全面的理解，并揭示在这一领域中通信高效解决方案的有希望的未来研究方向。