Stochastic gradient descent-based algorithms are widely used for training
deep neural networks but often suffer from slow convergence. To address the
challenge, we leverage the framework of the alternating direction method of
multipliers (ADMM) to develop a novel data-driven algorithm, called batch ADMM
(BADM). The fundamental idea of the proposed algorithm is to split the training
data into batches, which is further divided into sub-batches where primal and
dual variables are updated to generate global parameters through aggregation.
We evaluate the performance of BADM across various deep learning tasks,
including graph modelling, computer vision, image generation, and natural
language processing. Extensive numerical experiments demonstrate that BADM
achieves faster convergence and superior testing accuracy compared to other
state-of-the-art optimizers.

基于随机梯度下降的算法用于训练深度神经网络，但通常收敛速度较慢。为了解决这个挑战，我们利用交替方向乘子法（ADMM）的框架开发了一种新的数据驱动算法，称为批量 ADMM（BADM）。该算法的基本思想是将训练数据划分为批次，进一步将其分成子批次，在此过程中通过聚合更新原始和对偶变量以生成全局参数。我们评估了 BADM 在各种深度学习任务中的性能，包括图建模、计算机视觉、图像生成和自然语言处理。广泛的数值实验表明，与其他先进优化器相比，BADM 实现了更快的收敛速度和优越的测试准确性。

BADM: 深度学习的批量 ADMM

BADM: Batch ADMM for Deep Learning

We propose a statistical adaptive procedure called SALSA for automatically
scheduling the learning rate (step size) in stochastic gradient methods. SALSA
first uses a smoothed stochastic line-search procedure to gradually increase
the learning rate, then automatically switches to a statistical method to
decrease the learning rate. The line search procedure ``warms up'' the
optimization process, reducing the need for expensive trial and error in
setting an initial learning rate. The method for decreasing the learning rate
is based on a new statistical test for detecting stationarity when using a
constant step size. Unlike in prior work, our test applies to a broad class of
stochastic gradient algorithms without modification. The combined method is
highly robust and autonomous, and it matches the performance of the best
hand-tuned learning rate schedules in our experiments on several deep learning
tasks.

提出一种名为 SALSA 的统计自适应程序，用于自动调整随机梯度方法中的学习率（步长），该方法使用平滑的随机线性搜索程序逐渐增加学习率，然后自动转换为一个新的统计方法来降低学习率，它在广泛的随机梯度算法中使用了一个新的统计测试来检测状态，能够适应多类深度学习任务。