The emergence of Large Language Models (LLMs) has necessitated the adoption
of parallel training techniques, involving the deployment of thousands of GPUs
to train a single model. Unfortunately, we have found that the efficiency of
current parallel training is often suboptimal, largely due to the following two
main issues. Firstly, hardware failures are inevitable, leading to
interruptions in the training tasks. The inability to quickly identify the
faulty components results in a substantial waste of GPU resources. Secondly,
since GPUs must wait for parameter synchronization to complete before
proceeding to the next round of computation, network congestions can greatly
increase the waiting time for GPUs. To address these challenges, this paper
introduces a communication-driven solution, namely the C4. The key insights of
C4 are two folds. First, in parallel training, collective communication
exhibits periodic and homogeneous characteristics, so any anomalies are
certainly due to some form of hardware malfunction. By leveraging this feature,
C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
restart the task, thereby avoiding resource wastage caused by delays in anomaly
detection. Second, the predictable communication model of collective
communication, involving few large flows, allows C4 to efficiently execute
traffic planning, substantially reducing network congestion. C4 has been
extensively implemented across our production systems, cutting error-induced
overhead by roughly 30% and enhancing runtime performance by about 15% for
certain applications with moderate communication costs.

本文介绍了一种基于通信的解决方案 C4，通过利用并行训练中集体通信的周期性和均匀性特征，快速识别故障组件，避免了资源浪费，并且通过有效的流量规划大大降低了网络拥塞。该解决方案在生产系统中得到了广泛实施，削减了因错误导致的开销约 30％，并提高了某些应用程序的运行时性能约 15％。