Training large language models (LLMs) is known to be challenging because of
the huge computational and memory capacity requirements. To address these
issues, it is common to use a cluster of GPUs with 3D parallelism, which splits
a model along the data batch, pipeline stage, and intra-layer tensor
dimensions. However, the use of 3D parallelism produces the additional
challenge of finding the optimal number of ways on each dimension and mapping
the split models onto the GPUs. Several previous studies have attempted to
automatically find the optimal configuration, but many of these lacked several
important aspects. For instance, the heterogeneous nature of the interconnect
speeds is often ignored. While the peak bandwidths for the interconnects are
usually made equal, the actual attained bandwidth varies per link in real-world
clusters. Combined with the critical path modeling that does not properly
consider the communication, they easily fall into sub-optimal configurations.
In addition, they often fail to consider the memory requirement per GPU, often
recommending solutions that could not be executed. To address these challenges,
we propose Pipette, which is an automatic fine-grained LLM training
configurator for real-world clusters. By devising better performance models
along with the memory estimator and fine-grained individual GPU assignment,
Pipette achieves faster configurations that satisfy the memory constraints. We
evaluated Pipette on large clusters to show that it provides a significant
speedup over the prior art. The implementation of Pipette is available at
this https URL

Pipette 是一个针对现实世界集群的自动细粒度 LLM 训练配置器，通过设计更好的性能模型、内存估计器和细粒度的个别 GPU 分配，Pipette 在满足内存约束条件下实现更快的配置，并且相比现有技术提供了显著的加速。