Peer-to-peer deep learning algorithms are enabling distributed edge devices
to collaboratively train deep neural networks without exchanging raw training
data or relying on a central server. Peer-to-Peer Learning (P2PL) and other
algorithms based on Distributed Local-Update Stochastic/mini-batch Gradient
Descent (local DSGD) rely on interleaving epochs of training with distributed
consensus steps. This process leads to model parameter drift/divergence amongst
participating devices in both IID and non-IID settings. We observe that model
drift results in significant oscillations in test performance evaluated after
local training and consensus phases. We then identify factors that amplify
performance oscillations and demonstrate that our novel approach, P2PL with
Affinity, dampens test performance oscillations in non-IID settings without
incurring any additional communication cost.

使用点对点深度学习算法，实现分布式边缘设备之间的协作训练深度神经网络，无需交换原始训练数据或依赖中央服务器。我们发现模型漂移导致本地训练和共识阶段之后的测试性能出现显著震荡，并确定了放大性能震荡的因素。通过我们的创新方法 P2PL with Affinity，在非独立同分布的设置中，我们能够减弱测试性能震荡，而不增加额外的通信成本。