As modern DNN models grow ever larger, collective communications between the
accelerators (allreduce, etc.) emerge as a significant performance bottleneck.
Designing efficient communication schedules is challenging given today's highly
diverse and heterogeneous network fabrics. In this paper, we present
ForestColl, a tool that generates efficient schedules for any network topology.
ForestColl constructs broadcast/aggregation spanning trees as the communication
schedule, achieving theoretically minimum network congestion. Its schedule
generation runs in strongly polynomial time and is highly scalable. ForestColl
supports any network fabrics, including both switching fabrics and direct
connections, as well as any network graph structure. We evaluated ForestColl on
multi-cluster AMD MI250 and NVIDIA A100 platforms. ForestColl's schedules
achieved up to 52\% higher performance compared to the vendors' own optimized
communication libraries, RCCL and NCCL. ForestColl also outperforms other
state-of-the-art schedule generation techniques with both up to 61\% more
efficient generated schedules and orders of magnitude faster schedule
generation speed.

在现代 DNN 模型不断增大的同时，加速器之间的集体通信（allreduce 等）成为一个重要的性能瓶颈。在今天高度多样化和异构的网络结构下，设计高效的通信调度是具有挑战性的。本文提出了 ForestColl，一种为任何网络拓扑生成高效调度的工具。ForestColl 构建广播 / 聚合的生成树作为通信调度，实现了理论上的最小网络拥塞。其调度生成在强多项式时间内运行，并具有高度可扩展性。在多集群 AMD MI250 和 NVIDIA A100 平台上评估了 ForestColl。与供应商自己优化的通信库 RCCL 和 NCCL 相比，ForestColl 的调度性能提高了多达 52％。ForestColl 还在生成调度效率上超过其他最先进的调度生成技术，生成的调度更加高效，在调度生成速度上有数量级的提升。

ForestColl：异构网络纺织品上的高效集体通信

ForestColl: Efficient Collective Communications on Heterogeneous Network  Fabrics

Collective communications are an indispensable part of distributed training.
Running a topology-aware collective algorithm is crucial for optimizing
communication performance by minimizing congestion. Today such algorithms only
exist for a small set of simple topologies, limiting the topologies employed in
training clusters and handling irregular topologies due to network failures. In
this paper, we propose TACOS, an automated topology-aware collective
synthesizer for arbitrary input network topologies. TACOS synthesized 3.73x
faster All-Reduce algorithm over baselines, and synthesized collective
algorithms for 512-NPU system in just 6.1 minutes.

本文介绍了 TACOS，一种自动化的拓扑感知集体算法合成器，可应用于任意输入网络拓扑结构，优化了通信效率，加速了 All-Reduce 算法，适用于分布式训练等领域。