Modern Deep Learning (DL) models have grown to sizes requiring massive clusters of specialized, high-end nodes to train. Designing such clusters to maximize both performance and utilization to amortize their steep cost is a challenging task requiring careful balance of compute, memory, and network resources. Moreover, a plethora of each model's tuning knobs drastically affect the performance, with optimal values often depending on the underlying cluster's characteristics, which necessitates a complex cluster-workload co-design process. To facilitate the design space exploration of such massive DL training clusters, we introduce COMET a holistic cluster design methodology and workflow to jointly study the impact of parallelization strategies and key cluster resource provisioning on the performance of distributed DL training. We develop a step-by-step process to establish a reusable and flexible methodology, and demonstrate its application with a case study of training a Transformer-1T model on a cluster of variable compute, memory, and network resources. Our case study demonstrates COMET's utility in identifying promising architectural optimization directions and guiding system designers in configuring key model and cluster parameters.

介绍了一种名为COMET的深度学习集群设计方法和工作流程，该方法能够同时研究并优化并行化策略和关键集群资源配置以提高分布式深度学习模型的性能。纵观一个变量大小的计算，内存和网络资源的集群上训练基于Transformer-1T模型的实例，这个方法能够指导系统设计者进行架构优化方向的筛选，并协助其配置关键模型和集群参数。

COMET：分布式深度学习训练的综合集群设计方法