As modern DNN models grow ever larger, collective communications between the
accelerators (allreduce, etc.) emerge as a significant performance bottleneck.
Designing efficient communication schedules is challenging given today's highly
diverse and heterogeneous network fabrics. In this paper, we present
ForestColl, a tool that generates efficient schedules for any network topology.
ForestColl constructs broadcast/aggregation spanning trees as the communication
schedule, achieving theoretically minimum network congestion. Its schedule
generation runs in strongly polynomial time and is highly scalable. ForestColl
supports any network fabrics, including both switching fabrics and direct
connections, as well as any network graph structure. We evaluated ForestColl on
multi-cluster AMD MI250 and NVIDIA A100 platforms. ForestColl's schedules
achieved up to 52\% higher performance compared to the vendors' own optimized
communication libraries, RCCL and NCCL. ForestColl also outperforms other
state-of-the-art schedule generation techniques with both up to 61\% more
efficient generated schedules and orders of magnitude faster schedule
generation speed.

在现代 DNN 模型不断增大的同时，加速器之间的集体通信（allreduce 等）成为一个重要的性能瓶颈。在今天高度多样化和异构的网络结构下，设计高效的通信调度是具有挑战性的。本文提出了 ForestColl，一种为任何网络拓扑生成高效调度的工具。ForestColl 构建广播 / 聚合的生成树作为通信调度，实现了理论上的最小网络拥塞。其调度生成在强多项式时间内运行，并具有高度可扩展性。在多集群 AMD MI250 和 NVIDIA A100 平台上评估了 ForestColl。与供应商自己优化的通信库 RCCL 和 NCCL 相比，ForestColl 的调度性能提高了多达 52％。ForestColl 还在生成调度效率上超过其他最先进的调度生成技术，生成的调度更加高效，在调度生成速度上有数量级的提升。