The emergence of Large Language Models (LLMs) has necessitated the adoption
of parallel training techniques, involving the deployment of thousands of GPUs
to train a single model. Unfortunately, we have found that the efficiency of
current parallel training is often suboptimal, largely due to the following two
main issues. Firstly, hardware failures are inevitable, leading to
interruptions in the training tasks. The inability to quickly identify the
faulty components results in a substantial waste of GPU resources. Secondly,
since GPUs must wait for parameter synchronization to complete before
proceeding to the next round of computation, network congestions can greatly
increase the waiting time for GPUs. To address these challenges, this paper
introduces a communication-driven solution, namely the C4. The key insights of
C4 are two folds. First, in parallel training, collective communication
exhibits periodic and homogeneous characteristics, so any anomalies are
certainly due to some form of hardware malfunction. By leveraging this feature,
C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
restart the task, thereby avoiding resource wastage caused by delays in anomaly
detection. Second, the predictable communication model of collective
communication, involving few large flows, allows C4 to efficiently execute
traffic planning, substantially reducing network congestion. C4 has been
extensively implemented across our production systems, cutting error-induced
overhead by roughly 30% and enhancing runtime performance by about 15% for
certain applications with moderate communication costs.

本文介绍了一种基于通信的解决方案 C4，通过利用并行训练中集体通信的周期性和均匀性特征，快速识别故障组件，避免了资源浪费，并且通过有效的流量规划大大降低了网络拥塞。该解决方案在生产系统中得到了广泛实施，削减了因错误导致的开销约 30％，并提高了某些应用程序的运行时性能约 15％。

使用 C4 提升大规模并行训练效率：一种基于通信的方法

Boosting Large-scale Parallel Training Efficiency with C4: A  Communication-Driven Approach

Given a similarity graph between items, correlation clustering (CC) groups
similar items together and dissimilar ones apart. One of the most popular CC
algorithms is KwikCluster: an algorithm that serially clusters neighborhoods of
vertices, and obtains a 3-approximation ratio. Unfortunately, KwikCluster in
practice requires a large number of clustering rounds, a potential bottleneck
for large graphs.
We present C4 and ClusterWild!, two algorithms for parallel correlation
clustering that run in a polylogarithmic number of rounds and achieve nearly
linear speedups, provably. C4 uses concurrency control to enforce
serializability of a parallel clustering process, and guarantees a
3-approximation ratio. ClusterWild! is a coordination free algorithm that
abandons consistency for the benefit of better scaling; this leads to a
provably small loss in the 3-approximation ratio.
We provide extensive experimental results for both algorithms, where we
outperform the state of the art, both in terms of clustering accuracy and
running time. We show that our algorithms can cluster billion-edge graphs in
under 5 seconds on 32 cores, while achieving a 15x speedup.

本文提出了 C4 和 ClusterWild！两个用于并行相关聚类的算法，并在实验中证明了它们在精度和运行时间方面的优势。