Split learning is a simple solution for Vertical Federated Learning (VFL), which has drawn substantial attention in both research and application due to its simplicity and efficiency. However, communication efficiency is still a crucial issue for split learning. In this paper, we investigate multiple communication reduction methods for split learning, including cut layer size reduction, top-k sparsification, quantization, and L1 regularization. Through analysis of the cut layer size reduction and top-k sparsification, we further propose randomized top-k sparsification, to make the model generalize and converge better. This is done by selecting top-k elements with a large probability while also having a small probability to select non-top-k elements. Empirical results show that compared with other communication-reduction methods, our proposed randomized top-k sparsification achieves a better model performance under the same compression level.

本文研究了多种通信压缩方法，包括切割层数减少、top-k限制、量化和L1正则化，并通过分析cut layer size reduction 和 top-k sparsification提出了随机化的top-k sparsification方法来提高模型泛化和收敛性，在相同的压缩程度下取得更好的模型性能。

随机Top-k稀疏化减少分布式学习的通信