Peer-to-peer deep learning algorithms are enabling distributed edge devices
to collaboratively train deep neural networks without exchanging raw training
data or relying on a central server. Peer-to-Peer Learning (P2PL) and other
algorithms based on Distributed Local-Update Stochastic/mini-batch Gradient
Descent (local DSGD) rely on interleaving epochs of training with distributed
consensus steps. This process leads to model parameter drift/divergence amongst
participating devices in both IID and non-IID settings. We observe that model
drift results in significant oscillations in test performance evaluated after
local training and consensus phases. We then identify factors that amplify
performance oscillations and demonstrate that our novel approach, P2PL with
Affinity, dampens test performance oscillations in non-IID settings without
incurring any additional communication cost.

使用点对点深度学习算法，实现分布式边缘设备之间的协作训练深度神经网络，无需交换原始训练数据或依赖中央服务器。我们发现模型漂移导致本地训练和共识阶段之后的测试性能出现显著震荡，并确定了放大性能震荡的因素。通过我们的创新方法 P2PL with Affinity，在非独立同分布的设置中，我们能够减弱测试性能震荡，而不增加额外的通信成本。

对等学习与非独立同分布数据的一致性

Peer-to-Peer Learning + Consensus with Non-IID Data

We present P2PL, a practical multi-device peer-to-peer deep learning
algorithm that, unlike the federated learning paradigm, does not require
coordination from edge servers or the cloud. This makes P2PL well-suited for
the sheer scale of beyond-5G computing environments like smart cities that
otherwise create range, latency, bandwidth, and single point of failure issues
for federated approaches.
P2PL introduces max norm synchronization to catalyze training, retains
on-device deep model training to preserve privacy, and leverages local
inter-device communication to implement distributed consensus. Each device
iteratively alternates between two phases: 1) on-device learning and 2)
distributed cooperation where they combine model parameters with nearby
devices. We empirically show that all participating devices achieve the same
test performance attained by federated and centralized training -- even with
100 devices and relaxed singly stochastic consensus weights. We extend these
experimental results to settings with diverse network topologies, sparse and
intermittent communication, and non-IID data distributions.

我们提出了 P2PL 算法，它是一种实用的多设备对等深度学习算法，不需要边缘服务器或云端的协调，适用于超过 5G 计算环境（如智能城市）中的规模、延迟、带宽和单点故障问题，并通过最大范数同步、设备上的深度模型训练保护隐私、本地设备间通信实现分布式一致性。通过实验证明，即使使用 100 台设备和松散的随机一致性权重，所有参与设备都能达到联邦和集中式训练所达到的相同测试性能，并将这些实验结果扩展到具有多样化网络拓扑、稀疏和间歇通信以及非 IID 数据分布的环境中。