The past few years have witnessed the flourishing of large-scale deep neural
network models with ever-growing parameter numbers. Training such large-scale
models typically requires massive memory and computing resources that exceed
those of a single GPU, necessitating distributed training. As GPU performance
has rapidly evolved in recent years, computation time has shrunk, thereby
increasing the proportion of communication in the overall training time.
Therefore, optimizing communication for distributed training has become an
urgent issue. In this article, we briefly introduce the general architecture of
distributed deep neural network training and analyze relationships among
Parallelization Strategy, Collective Communication Library, and Network from
the perspective of communication optimization, which forms a three-layer
paradigm. We then review current representative research advances with this
three-layer paradigm. We find that layers in the current three-layer paradigm
are relatively independent, but there is a rich design space for cross-layer
collaborative optimization in distributed training scenarios. Therefore, we
further advocate a communication-efficient five-layer paradigm underlining
opportunities for collaboration designs and look forward to the perspectives of
"Vertical", "Horizontal", "Intra-Inter" and "Host-Net" collaboration designs.
We hope this article can shed some light on future research on communication
optimization for distributed training.

大规模深度神经网络的分布式训练通信优化的研究，提出了一个五层范式，强调协同设计的机会。