We develop a novel framework that adds the regularizers of the sparse group
lasso to a family of adaptive optimizers in deep learning, such as Momentum,
Adagrad, Adam, AMSGrad, AdaHessian, and create a new class of optimizers, which
are named Group Momentum, Group Adagrad, Group Adam, Group AMSGrad and Group
AdaHessian, etc., accordingly. We establish theoretically proven convergence
guarantees in the stochastic convex settings, based on primal-dual methods. We
evaluate the regularized effect of our new optimizers on three large-scale
real-world ad click datasets with state-of-the-art deep learning models. The
experimental results reveal that compared with the original optimizers with the
post-processing procedure which uses the magnitude pruning method, the
performance of the models can be significantly improved on the same sparsity
level. Furthermore, in comparison to the cases without magnitude pruning, our
methods can achieve extremely high sparsity with significantly better or highly
competitive performance.

本文提出了一种将稀疏组套索的正则化器添加到深度学习中的一类自适应优化器中，经过实验验证在同等稀疏度下模型性能得到了显著提升。该方法在广告点击数据集上应用效果显著。

应用稀疏分组 Lasso 的自适应优化器在 CTR 预测神经网络中的应用

Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR  Prediction

We analyze the convergence rate of the alternating direction method of
multipliers (ADMM) for minimizing the sum of two or more nonsmooth convex
separable functions subject to linear constraints. Previous analysis of the
ADMM typically assumes that the objective function is the sum of only two
convex functions defined on two separable blocks of variables even though the
algorithm works well in numerical experiments for three or more blocks.
Moreover, there has been no rate of convergence analysis for the ADMM without
strong convexity in the objective function. In this paper we establish the
global linear convergence of the ADMM for minimizing the sum of any number of
convex separable functions. This result settles a key question regarding the
convergence of the ADMM when the number of blocks is more than two or if the
strong convexity is absent. It also implies the linear convergence of the ADMM
for several contemporary applications including LASSO, Group LASSO and Sparse
Group LASSO without any strong convexity assumption. Our proof is based on
estimating the distance from a dual feasible solution to the optimal dual
solution set by the norm of a certain proximal residual, and by requiring the
dual stepsize to be sufficiently small.

本文研究交替方向乘子法 (ADMM) 用于多个非光滑凸可分函数的线性约束约束下极小化问题的收敛速率，通过引入一种新的与其它满足该问题的近似算法有所不同的证明手段，我们在不限制强凸性的情况下，建立了全局线性收敛性的证明方案，表明 ADMM 的线性收敛性可以在三个以上的可分函数的情况下适用，包括 LASSO，Group LASSO 和 Sparse Group LASSO 等当代应用。