With rapidly increasing distributed deep learning workloads in large-scale
data centers, efficient distributed deep learning framework strategies for
resource allocation and workload scheduling have become the key to
high-performance deep learning. The large-scale environment with large volumes
of datasets, models, and computational and communication resources raises
various unique challenges for resource allocation and workload scheduling in
distributed deep learning, such as scheduling complexity, resource and workload
heterogeneity, and fault tolerance. To uncover these challenges and
corresponding solutions, this survey reviews the literature, mainly from 2019
to 2024, on efficient resource allocation and workload scheduling strategies
for large-scale distributed DL. We explore these strategies by focusing on
various resource types, scheduling granularity levels, and performance goals
during distributed training and inference processes. We highlight critical
challenges for each topic and discuss key insights of existing technologies. To
illustrate practical large-scale resource allocation and workload scheduling in
real distributed deep learning scenarios, we use a case study of training large
language models. This survey aims to encourage computer science, artificial
intelligence, and communications researchers to understand recent advances and
explore future research directions for efficient framework strategies for
large-scale distributed deep learning.

大规模分布式深度学习中，资源分配和工作负载调度的高效策略是实现高性能深度学习的关键。本综述调查了 2019 年至 2024 年间关于大规模分布式深度学习的资源分配和工作负载调度策略的文献，并通过探索不同类型的资源、调度粒度和性能目标，以及实际案例研究为例，突出了每个主题的关键挑战和现有技术的重要见解，旨在鼓励计算机科学、人工智能和通信研究人员了解最新进展并探索大规模分布式深度学习的未来研究方向。