In this paper, we concentrate on decentralized optimization problems with
nonconvex and nonsmooth objective functions, especially on the decentralized
training of nonsmooth neural networks. We introduce a unified framework, named
DSM, to analyze the global convergence of decentralized stochastic subgradient
methods. We prove the global convergence of our proposed framework under mild
conditions, by establishing that the generated sequence asymptotically
approximates the trajectories of its associated differential inclusion.
Furthermore, we establish that our proposed framework encompasses a wide range
of existing efficient decentralized subgradient methods, including
decentralized stochastic subgradient descent (DSGD), DSGD with
gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGDm). In
addition, we introduce SignSGD employing the sign map to regularize the update
directions in DSGDm, and show it is enclosed in our proposed framework.
Consequently, our convergence results establish, for the first time, global
convergence of these methods when applied to nonsmooth nonconvex objectives.
Preliminary numerical experiments demonstrate that our proposed framework
yields highly efficient decentralized subgradient methods with convergence
guarantees in the training of nonsmooth neural networks.

我们集中研究具有非凸非光滑目标函数的分布式优化问题，特别是非光滑神经网络的分布式训练。我们引入一个统一框架，名为 DSM，用于分析分布式随机次梯度方法的全局收敛性。我们证明了在温和条件下，我们提出的框架的全局收敛性，通过建立生成的序列渐近逼近其相关微分纳入的轨迹。此外，我们证明了我们提出的框架包括各种现有的高效分布式次梯度方法，包括分布式随机次梯度下降（DSGD），带有梯度跟踪技术的 DSGD（DSGD-T）和带有动量的 DSGD（DSGDm）。此外，我们引入 SignSGD，使用符号映射来规范 DSGDm 中的更新方向，并证明其包含在我们提出的框架中。因此，我们的收敛结果首次证明了这些方法在应用于非光滑非凸目标时的全局收敛性。初步的数值实验表明，我们提出的框架在非光滑神经网络的训练中产生了高效的分布式次梯度方法，具有收敛性保证。

去中心化随机次梯度法用于非光滑非凸优化

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex  Optimization

In this paper, we investigate the convergence properties of a wide class of
Adam-family methods for minimizing quadratically regularized nonsmooth
nonconvex optimization problems, especially in the context of training
nonsmooth neural networks with weight decay. Motivated by the AdamW method, we
propose a novel framework for Adam-family methods with decoupled weight decay.
Within our framework, the estimators for the first-order and second-order
moments of stochastic subgradients are updated independently of the weight
decay term. Under mild assumptions and with non-diminishing stepsizes for
updating the primary optimization variables, we establish the convergence
properties of our proposed framework. In addition, we show that our proposed
framework encompasses a wide variety of well-known Adam-family methods, hence
offering convergence guarantees for these methods in the training of nonsmooth
neural networks. More importantly, we show that our proposed framework
asymptotically approximates the SGD method, thereby providing an explanation
for the empirical observation that decoupled weight decay enhances
generalization performance for Adam-family methods. As a practical application
of our proposed framework, we propose a novel Adam-family method named Adam
with Decoupled Weight Decay (AdamD), and establish its convergence properties
under mild conditions. Numerical experiments demonstrate that AdamD outperforms
Adam and is comparable to AdamW, in the aspects of both generalization
performance and efficiency.

本研究探讨了一类广泛的 Adam 系列方法在最小化二次正则化非光滑非凸优化问题中的收敛性质，特别是在训练带有权重衰减的非光滑神经网络的情况下。通过引入修正权重衰减的 AdamW 方法，我们提出了一种新颖的具有分离权重衰减的 Adam 系列方法的框架。在这个框架中，对于随机子梯度的一阶和二阶矩的估计独立于权重衰减项进行更新。在适当的假设和非缩减的步长下，我们证明了所提出框架的收敛性质。此外，我们还表明所提出的框架包含了多种众所周知的 Adam 系列方法，从而为这些方法在训练非光滑神经网络时提供了收敛性保证。更重要的是，我们展示了所提出的框架渐近逼近 SGD 方法，从而解释了独立权重衰减如何增强 Adam 系列方法的泛化性能的经验观察。作为我们提出框架的实际应用，我们提出了一种名为 Adam with Decoupled Weight Decay (AdamD) 的新型 Adam 系列方法，并在适当条件下证明了其收敛性质。数值实验表明，AdamD 在泛化性能和效率方面优于 Adam，并与 AdamW 相当。

深度学习中的分离权重衰减的 Adam 系列方法

Adam-family Methods with Decoupled Weight Decay in Deep Learning

In this paper, we present a comprehensive study on the convergence properties
of Adam-family methods for nonsmooth optimization, especially in the training
of nonsmooth neural networks. We introduce a novel two-timescale framework that
adopts a two-timescale updating scheme, and prove its convergence properties
under mild assumptions. Our proposed framework encompasses various popular
Adam-family methods, providing convergence guarantees for these methods in
training nonsmooth neural networks. Furthermore, we develop stochastic
subgradient methods that incorporate gradient clipping techniques for training
nonsmooth neural networks with heavy-tailed noise. Through our framework, we
show that our proposed methods converge even when the evaluation noises are
only assumed to be integrable. Extensive numerical experiments demonstrate the
high efficiency and robustness of our proposed methods.

研究了 Adam 族方法在非光滑优化尤其是在训练非光滑神经网络方面的收敛性，提出了一个采用双时间尺度更新方案的新颖框架并证明了其在温和假设下的收敛性，介绍了一些植入了梯度剪切技术的随机次梯度方法并通过实验证明其高效性和健壮性。