In this paper, we concentrate on decentralized optimization problems with nonconvex and nonsmooth objective functions, especially on the decentralized training of nonsmooth neural networks. We introduce a unified framework, named DSM, to analyze the global convergence of decentralized stochastic subgradient methods. We prove the global convergence of our proposed framework under mild conditions, by establishing that the generated sequence asymptotically approximates the trajectories of its associated differential inclusion. Furthermore, we establish that our proposed framework encompasses a wide range of existing efficient decentralized subgradient methods, including decentralized stochastic subgradient descent (DSGD), DSGD with gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGDm). In addition, we introduce SignSGD employing the sign map to regularize the update directions in DSGDm, and show it is enclosed in our proposed framework. Consequently, our convergence results establish, for the first time, global convergence of these methods when applied to nonsmooth nonconvex objectives. Preliminary numerical experiments demonstrate that our proposed framework yields highly efficient decentralized subgradient methods with convergence guarantees in the training of nonsmooth neural networks.

我们集中研究具有非凸非光滑目标函数的分布式优化问题，特别是非光滑神经网络的分布式训练。我们引入一个统一框架，名为DSM，用于分析分布式随机次梯度方法的全局收敛性。我们证明了在温和条件下，我们提出的框架的全局收敛性，通过建立生成的序列渐近逼近其相关微分纳入的轨迹。此外，我们证明了我们提出的框架包括各种现有的高效分布式次梯度方法，包括分布式随机次梯度下降（DSGD），带有梯度跟踪技术的DSGD（DSGD-T）和带有动量的DSGD（DSGDm）。此外，我们引入SignSGD，使用符号映射来规范DSGDm中的更新方向，并证明其包含在我们提出的框架中。因此，我们的收敛结果首次证明了这些方法在应用于非光滑非凸目标时的全局收敛性。初步的数值实验表明，我们提出的框架在非光滑神经网络的训练中产生了高效的分布式次梯度方法，具有收敛性保证。

去中心化随机次梯度法用于非光滑非凸优化