The stochastic gradient descent (SGD) algorithm has achieved remarkable success in training deep learning models. However, it has several limitations, including susceptibility to vanishing gradients, sensitivity to input data, and a lack of robust theoretical guarantees. In recent years, alternating minimization (AM) methods have emerged as a promising alternative for model training by employing gradient-free approaches to iteratively update model parameters. Despite their potential, these methods often exhibit slow convergence rates. To address this challenge, we propose a novel Triple-Inertial Accelerated Alternating Minimization (TIAM) framework for neural network training. The TIAM approach incorporates a triple-inertial acceleration strategy with a specialized approximation method, facilitating targeted acceleration of different terms in each sub-problem optimization. This integration improves the efficiency of convergence, achieving superior performance with fewer iterations. Additionally, we provide a convergence analysis of the TIAM algorithm, including its global convergence properties and convergence rate. Extensive experiments validate the effectiveness of the TIAM method, showing significant improvements in generalization capability and computational efficiency compared to existing approaches, particularly when applied to the rectified linear unit (ReLU) and its variants.

本研究针对随机梯度下降算法在深度学习模型训练中存在的梯度消失和收敛速度慢等问题，提出了一种新的三重惯性加速交替最小化框架（TIAM）。该方法通过引入三重惯性加速策略和特殊的近似方法，有效提高了模型训练的收敛效率，实验证明其在普适性和计算效率上均显著优于现有方法，尤其在使用修正线性单元及其变体时表现突出。

深度学习训练的三重惯性加速交替优化方法