Decentralized learning (DL) systems have been gaining popularity because they
avoid raw data sharing by communicating only model parameters, hence preserving
data confidentiality. However, the large size of deep neural networks poses a
significant challenge for decentralized training, since each node needs to
exchange gigabytes of data, overloading the network. In this paper, we address
this challenge with JWINS, a communication-efficient and fully decentralized
learning system that shares only a subset of parameters through sparsification.
JWINS uses wavelet transform to limit the information loss due to
sparsification and a randomized communication cut-off that reduces
communication usage without damaging the performance of trained models. We
demonstrate empirically with 96 DL nodes on non-IID datasets that JWINS can
achieve similar accuracies to full-sharing DL while sending up to 64% fewer
bytes. Additionally, on low communication budgets, JWINS outperforms the
state-of-the-art communication-efficient DL algorithm CHOCO-SGD by up to 4x in
terms of network savings and time.

本文介绍了 JWINS，一种通信高效且完全分散的学习系统，通过稀疏化仅共享少量参数，使用小波变换和随机通信切断降低信息损失并减少传输字节数，可在非 I ID 数据集上训练出接近于全共享的 DL 准确率，且可节省多达 64% 的字节数，在低通信预算下优于 CHOCO-SGD 算法。