We analyze the effect of synchronization on distributed stochastic gradient
algorithms. By exploiting an analogy with dynamical models of biological quorum
sensing - where synchronization between agents is induced through communication
with a common signal - we quantify how synchronization can significantly reduce
the magnitude of the noise felt by the individual distributed agents and by
their spatial mean. This noise reduction is in turn associated with a reduction
in the smoothing of the loss function imposed by the stochastic gradient
approximation. Through simulations on model non-convex objectives, we
demonstrate that coupling can stabilize higher noise levels and improve
convergence. We provide a convergence analysis for strongly convex functions by
deriving a bound on the expected deviation of the spatial mean of the agents
from the global minimizer for an algorithm based on quorum sensing, the same
algorithm with momentum, and the Elastic Averaging SGD (EASGD) algorithm. We
discuss extensions to new algorithms that allow each agent to broadcast its
current measure of success and shape the collective computation accordingly. We
supplement our theoretical analysis with numerical experiments on convolutional
neural networks trained on the CIFAR-10 dataset, where we note a surprising
regularizing property of EASGD even when applied to the non-distributed case.
This observation suggests alternative second-order in-time algorithms for
non-distributed optimization that are competitive with momentum methods.

分析了同步对分布式随机梯度算法的影响。通过模拟生物群体感应模型，量化了同步如何显着减少分布式代理和它们的空间平均值感受到的噪声幅度，从而降低了随机梯度逼近所施加的损失函数的平滑性，并通过对模型非凸目标的模拟，证明耦合可以稳定更高的噪声水平并提高收敛性。

分布式随机梯度的连续时间分析

A continuous-time analysis of distributed stochastic gradient

In many natural synchronization phenomena, communication between individual
elements occurs not directly, but rather through the environment. One of these
instances is bacterial quorum sensing, where bacteria release signaling
molecules in the environment which in turn are sensed and used for population
coordination. Extending this motivation to a general non- linear dynamical
system context, this paper analyzes synchronization phenomena in networks where
communication and coupling between nodes are mediated by shared dynamical quan-
tities, typically provided by the nodes' environment. Our model includes the
case when the dynamics of the shared variables themselves cannot be neglected
or indeed play a central part. Applications to examples from systems biology
illustrate the approach.

本文将细菌族群感应中介质传递的现象拓展到非线性动力系统的组织中，分析节点间通过共享动态量进行交流和耦合的同步现象，并给出了系统生物学中的实现例子。