We propose a novel GPU-cluster scheduler for distributed DL (DDL) workloads
that enables proximity based consolidation of GPU resources based on the DDL
jobs' sensitivities to the anticipated communication-network delays. Our
scheduler consists of three major components: (i) a classical delay scheduling
algorithm to facilitate job placement and consolidation; (ii) a
network-sensitive job preemption strategy; and (iii) an "auto-tuner" mechanism
to optimize delay timers for effective delay scheduling. Additionally, to
enable a cost-effective methodology for large-scale experiments, we develop a
data-driven DDL cluster simulation platform. Employing the simulation platform
we compare against several state-of-the-art alternatives on real-world workload
traces to demonstrate the benefits of our design. Our scheduler can provide
improvement of up to 69% in end-to-end Makespan for training all jobs compared
to the prevailing consolidation-based scheduling methods, while reducing the
average job completion time by up to 83% and minimizing the communication
overheads by up to 98% under congested networking conditions.

我们提出了一种新颖的 GPU 集群调度器，用于分布式深度学习工作负载，基于 DDL 作业对预期通信网络延迟的敏感性实现了基于邻近性的资源整合，通过经典的延迟调度算法来实现作业的放置和整合，使用数据驱动的 DDL 集群模拟平台进行了大规模实验，证明了我们设计的优势。在拥挤的网络条件下，与现有的基于整合的调度方法相比，我们的调度器可以提供多达 69% 的端到端完成时间的改进，同时将平均作业完成时间降低多达 83%，并将通信开销最小化多达 98%。