Decentralized training enables learning with distributed datasets generated
at different locations without relying on a central server. In realistic
scenarios, the data distribution across these sparsely connected learning
agents can be significantly heterogeneous, leading to local model over-fitting
and poor global model generalization. Another challenge is the high
communication cost of training models in such a peer-to-peer fashion without
any central coordination. In this paper, we jointly tackle these two-fold
practical challenges by proposing SADDLe, a set of sharpness-aware
decentralized deep learning algorithms. SADDLe leverages Sharpness-Aware
Minimization (SAM) to seek a flatter loss landscape during training, resulting
in better model generalization as well as enhanced robustness to communication
compression. We present two versions of our approach and conduct extensive
experiments to show that SADDLe leads to 1-20% improvement in test accuracy
compared to other existing techniques. Additionally, our proposed approach is
robust to communication compression, with an average drop of only 1% in the
presence of up to 4x compression.

本文提出 SADDLe，一种利用 Sharpness-Aware Minimization (SAM) 的锐度感知的分布式深度学习算法，以解决分布式训练中的局部过拟合、全局模型泛化不佳和高通信成本等实际挑战。实验结果表明，与现有技术相比，SADDLe 能使测试准确率提高 1-20%，且在 4 倍的通信压缩下，仅平均下降 1%。