We propose an efficient protocol for decentralized training of deep neural
networks from distributed data sources. The proposed protocol allows to handle
different phases of model training equally well and to quickly adapt to concept
drifts. This leads to a reduction of communication by an order of magnitude
compared to periodically communicating state-of-the-art approaches. Moreover,
we derive a communication bound that scales well with the hardness of the
serialized learning problem. The reduction in communication comes at almost no
cost, as the predictive performance remains virtually unchanged. Indeed, the
proposed protocol retains loss bounds of periodically averaging schemes. An
extensive empirical evaluation validates major improvement of the trade-off
between model performance and communication which could be beneficial for
numerous decentralized learning applications, such as autonomous driving, or
voice recognition and image classification on mobile phones.

我们提出了一种高效的协议，用于分布式数据源的分散式深度神经网络训练，该协议允许同等处理模型训练的不同阶段，并快速适应概念偏移，这导致与周期性通信的最先进的方法相比，通信减少了一个数量级。此外，我们得出一个通信限制，该限制随序列化学习问题难度的增加而缩放得很好，通信的减少几乎没有代价，因为预测性能保持几乎不变。实验证实了模型性能和通信之间的权衡的显着改进，这可能有助于许多分散式学习应用，例如自动驾驶或移动电话上的语音识别和图像分类。

动态模型平均的高效分散式深度学习

Efficient Decentralized Deep Learning by Dynamic Model Averaging

When applying aggregating strategies to Prediction with Expert Advice, the
learning rate must be adaptively tuned. The natural choice of
sqrt(complexity/current loss) renders the analysis of Weighted Majority
derivatives quite complicated. In particular, for arbitrary weights there have
been no results proven so far. The analysis of the alternative "Follow the
Perturbed Leader" (FPL) algorithm from Kalai & Vempala (2003) (based on
Hannan's algorithm) is easier. We derive loss bounds for adaptive learning rate
and both finite expert classes with uniform weights and countable expert
classes with arbitrary weights. For the former setup, our loss bounds match the
best known results so far, while for the latter our results are new.

应用聚合策略进行预测时，需要自适应调整学习速率以避免复杂度和当前损失率之间的分析难题；本文基于 Kalai 和 Vempala（2003）的 “Follow the Perturbed Leader”（FPL）算法，在两种不同的专家类别下得出了可调学习速率的损失界限，其中前者的损失界限与迄今为止最佳结果匹配，而后者为新结果。