Rapid advances in artificial intelligence (AI) technology have led to
significant accuracy improvements in a myriad of application domains at the
cost of larger and more compute-intensive models. Training such models on
massive amounts of data typically requires scaling to many compute nodes and
relies heavily on collective communication algorithms, such as all-reduce, to
exchange the weight gradients between different nodes. The overhead of these
collective communication operations in a distributed AI training system can
bottleneck its performance, with more pronounced effects as the number of nodes
increases. In this paper, we first characterize the all-reduce operation
overhead by profiling distributed AI training. Then, we propose a new smart
network interface card (NIC) for distributed AI training systems using
field-programmable gate arrays (FPGAs) to accelerate all-reduce operations and
optimize network bandwidth utilization via data compression. The AI smart NIC
frees up the system's compute resources to perform the more compute-intensive
tensor operations and increases the overall node-to-node communication
efficiency. We perform real measurements on a prototype distributed AI training
system comprised of 6 compute nodes to evaluate the performance gains of our
proposed FPGA-based AI smart NIC compared to a baseline system with regular
NICs. We also use these measurements to validate an analytical model that we
formulate to predict performance when scaling to larger systems. Our proposed
FPGA-based AI smart NIC enhances overall training performance by 1.6x at 6
nodes, with an estimated 2.5x performance improvement at 32 nodes, compared to
the baseline system using conventional NICs.

通过使用基于 FPGA 的 AI 智能网卡加速所有约减操作并通过数据压缩优化网络带宽利用率，该论文提出了一种新的分布式 AI 培训系统，为计算密集型张量操作释放了系统的计算资源，并提高了节点间通信效率。在验证分析模型的同时，该方法可将培训性能提高 1.6 倍至 6 个节点，而在 32 个节点时可能有 2.5 倍的性能提升。