Stochastic gradient descent-based algorithms are widely used for training deep neural networks but often suffer from slow convergence. To address the challenge, we leverage the framework of the alternating direction method of multipliers (ADMM) to develop a novel data-driven algorithm, called batch ADMM (BADM). The fundamental idea of the proposed algorithm is to split the training data into batches, which is further divided into sub-batches where primal and dual variables are updated to generate global parameters through aggregation. We evaluate the performance of BADM across various deep learning tasks, including graph modelling, computer vision, image generation, and natural language processing. Extensive numerical experiments demonstrate that BADM achieves faster convergence and superior testing accuracy compared to other state-of-the-art optimizers.

基于随机梯度下降的算法用于训练深度神经网络，但通常收敛速度较慢。为了解决这个挑战，我们利用交替方向乘子法（ADMM）的框架开发了一种新的数据驱动算法，称为批量ADMM（BADM）。该算法的基本思想是将训练数据划分为批次，进一步将其分成子批次，在此过程中通过聚合更新原始和对偶变量以生成全局参数。我们评估了BADM在各种深度学习任务中的性能，包括图建模、计算机视觉、图像生成和自然语言处理。广泛的数值实验表明，与其他先进优化器相比，BADM实现了更快的收敛速度和优越的测试准确性。

BADM: 深度学习的批量ADMM