Alternating Direction Method of Multipliers (ADMM) has been used successfully
in many conventional machine learning applications and is considered to be a
useful alternative to Stochastic Gradient Descent (SGD) as a deep learning
optimizer. However, as an emerging domain, several challenges remain, including
1) The lack of global convergence guarantees, 2) Slow convergence towards
solutions, and 3) Cubic time complexity with regard to feature dimensions. In
this paper, we propose a novel optimization framework for deep learning via
ADMM (dlADMM) to address these challenges simultaneously. The parameters in
each layer are updated backward and then forward so that the parameter
information in each layer is exchanged efficiently. The time complexity is
reduced from cubic to quadratic in (latent) feature dimensions via a dedicated
algorithm design for subproblems that enhances them utilizing iterative
quadratic approximations and backtracking. Finally, we provide the first proof
of global convergence for an ADMM-based method (dlADMM) in a deep neural
network problem under mild conditions. Experiments on benchmark datasets
demonstrated that our proposed dlADMM algorithm outperforms most of the
comparison methods.

本文提出 dlADMM 算法解决常规深度学习中使用 ADMM 所面临的全局收敛保证缺失，收敛速度慢以及计算时间复杂度高的问题，并且通过在子问题中使用迭代二次近似和回溯进行有效增强的特定算法设计，将时间复杂度从特征维度的立方降至二次，并在温和条件下提供了支持 ADMM 的深度神经网络问题全局收敛的第一份证明。大规模实验表明，本文所提出的 dlADMM 算法优于大多数比较方法。