Training and deploying large machine learning (ML) models is time-consuming
and requires significant distributed computing infrastructures. Based on
real-world large model training on datacenter-scale infrastructures, we show
14~32% of all GPU hours are spent on communication with no overlapping
computation. To minimize the outstanding communication latency, in this work,
we develop an agile performance modeling framework to guide parallelization and
hardware-software co-design strategies. Using the suite of real-world large ML
models on state-of-the-art GPU training hardware, we demonstrate 2.24x and
5.27x throughput improvement potential for pre-training and inference
scenarios, respectively.

基于真实世界的大规模模型训练和数据中心规模基础设施，我们展示了 14~32% 的 GPU 时间用于非重叠计算的通信。为了最小化未完成的通信延迟，在这项工作中，我们开发了一个灵活的性能建模框架来指导并行化和硬件软件共同设计策略。通过使用现代 GPU 训练硬件上的真实大规模 ML 模型套件，我们分别展示了预训练和推理场景的 2.24 倍和 5.27 倍的吞吐量提升潜力。