Alternating minimization methods have recently been proposed as alternatives to the gradient descent for deep neural network optimization. Alternating minimization methods can typically decompose a deep neural network into layerwise subproblems, which can then be optimized in parallel. Despite the significant parallelism, alternating minimization methods are rarely explored in training deep neural networks because of the severe accuracy degradation. In this paper, we analyze the reason and propose to achieve a compelling trade-off between parallelism and accuracy by a reformulation called Tunable Subnetwork Splitting Method (TSSM), which can tune the decomposition granularity of deep neural networks. Two methods gradient splitting Alternating Direction Method of Multipliers (gsADMM) and gradient splitting Alternating Minimization (gsAM) are proposed to solve the TSSM formulation. Experiments on five benchmark datasets show that our proposed TSSM can achieve significant speedup without observable loss of training accuracy. The code has been released at https://github.com/xianggebenben/TSSM.

本文提出了一种称为可调子网络分割方法（Tunable Subnetwork Splitting Method，TSSM）的重新方法，该方法可以在并行优化中实现深度神经网络的分解粒度的优化，以实现并行性和准确性之间的完美折衷。两种方法通过TSSM公式进行求解，并在五个基准数据集上进行实验证明，提出的TSSM可以显着提高速度，同时不会降低训练准确性。

神经网络训练的模型并行化调谐子网络分割