The past few years have witnessed growth in the size and computational requirements for training deep convolutional neural networks. Current approaches parallelize the training process onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, this design results in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our experiments show that layer-wise parallelism outperforms current parallelization approaches by increasing training speed, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining the same network accuracy.

本文提出了一种层级并行的方法来训练深度卷积神经网络，在这种方法中，每层神经网络都可以应用不同的并行策略以优化训练，通过解决图搜索问题来协同优化每个层的并行化方式。 实验结果表明，与现有的并行化方法相比，采用该方法可以提高训练吞吐量，降低通信成本，实现更好的多GPU可扩展性，并保持原始网络的准确性。

探索卷积神经网络并行化中的隐藏维度