This paper presents a new generalization error analysis for the Decentralized
Stochastic Gradient Descent (D-SGD) algorithm based on algorithmic stability.
The obtained results largely improve upon state-of-the-art results, and even
invalidate their claims that the communication graph has a detrimental effect
on generalization. For instance, we show that in convex settings, D-SGD has the
same generalization bounds as the classical SGD algorithm, no matter the choice
of graph. We exhibit that this counter-intuitive result comes from considering
the average of local parameters, which hides a final global averaging step
incompatible with the decentralized scenario. In light of this observation, we
advocate to analyze the supremum over local parameters and show that in this
case, the graph does have an impact on the generalization. Unlike prior
results, our analysis yields non-vacuous bounds even for non-connected graphs.

本文提出 Decentralized Stochastic Gradient Descent 算法的泛化误差分析，并据此证明在凸设置下，不论选择哪种通信图，D-SGD 算法的泛化界限与经典 SGD 算法相同，即前人论述的通信图对泛化的不利影响并不成立。

分散式 SGD 算法的稳定性和泛化分析改进

Improved Stability and Generalization Analysis of the Decentralized SGD  Algorithm

Understanding the convergence performance of asynchronous stochastic gradient
descent method (Async-SGD) has received increasing attention in recent years
due to their foundational role in machine learning. To date, however, most of
the existing works are restricted to either bounded gradient delays or convex
settings. In this paper, we focus on Async-SGD and its variant Async-SGDI
(which uses increasing batch size) for non-convex optimization problems with
unbounded gradient delays. We prove $o(1/\sqrt{k})$ convergence rate for
Async-SGD and $o(1/k)$ for Async-SGDI. Also, a unifying sufficient condition
for Async-SGD's convergence is established, which includes two major gradient
delay models in the literature as special cases and yields a new delay model
not considered thus far.

研究异步随机梯度下降法在无界梯度延迟的非凸优化问题中的收敛性能，证明了 Async-SGD 和其变体 Async-SGDI 的收敛速度，建立了一个异步随机梯度下降法的充分条件，并将两个梯度延迟模型作为其特殊情况包括在内，得出了一个新的未考虑的延迟模型。

非凸学习中无界时延异步随机梯度下降的收敛控制

Taming Convergence for Asynchronous Stochastic Gradient Descent with  Unbounded Delay in Non-Convex Learning

Distributed machine learning algorithms enable learning of models from
datasets that are distributed over a network without gathering the data at a
centralized location. While efficient distributed algorithms have been
developed under the assumption of faultless networks, failures that can render
these algorithms nonfunctional occur frequently in the real world. This paper
focuses on the problem of Byzantine failures, which are the hardest to
safeguard against in distributed algorithms. While Byzantine fault tolerance
has a rich history, existing work does not translate into efficient and
practical algorithms for high-dimensional learning in fully distributed (also
known as decentralized) settings. In this paper, an algorithm termed
Byzantine-resilient distributed coordinate descent (ByRDiE) is developed and
analyzed that enables distributed learning in the presence of Byzantine
failures. Theoretical analysis (convex settings) and numerical experiments
(convex and nonconvex settings) highlight its usefulness for high-dimensional
distributed learning in the presence of Byzantine failures.

本研究开发和分析了一种名为 ByRDiE 的算法，使得在分布式学习中存在 Byzantine 故障的情况下，能够高效地对高维数据集进行分布式学习，理论分析和数值实验说明了它在凸性和非凸性设置中的实用性。