Joint consideration of scheduling and adaptive parallelism offers great
opportunities for improving the training efficiency of large models on
heterogeneous GPU clusters. However, integrating adaptive parallelism into a
cluster scheduler expands the cluster scheduling space. The new space is the
product of the original scheduling space and the parallelism exploration space
of adaptive parallelism (also a product of pipeline, data, and tensor
parallelism). The exponentially enlarged scheduling space and ever-changing
optimal parallelism plan from adaptive parallelism together result in the
contradiction between low-overhead and accurate performance data acquisition
for efficient cluster scheduling. This paper presents Crius, a training system
for efficiently scheduling multiple large models with adaptive parallelism in a
heterogeneous cluster. Crius proposes a novel scheduling granularity called
Cell. It represents a job with deterministic resources and pipeline stages. The
exploration space of Cell is shrunk to the product of only data and tensor
parallelism, thus exposing the potential for accurate and low-overhead
performance estimation. Crius then accurately estimates Cells and efficiently
schedules training jobs. When a Cell is selected as a scheduling choice, its
represented job runs with the optimal parallelism plan explored. Experimental
results show that Crius reduces job completion time by up to 48.9% and
schedules large models with up to 1.49x cluster throughput improvement.

Crius 是一个用于在异构集群中以自适应并行性高效调度多个大型模型的训练系统，它通过引入称为 Cell 的新调度粒度将数据并行性和张量并行性的探索空间缩小到最小，从而实现准确且低开销的性能估计。通过选择 Cell 作为调度选择，Crius 能够运行具有优化并行性计划的作业，实验结果表明 Crius 能够将作业完成时间缩短高达 48.9%，并提高 1.49 倍的集群吞吐量。