Collaborative self-supervised learning has recently become feasible in highly
distributed environments by dividing the network layers between client devices
and a central server. However, state-of-the-art methods, such as MocoSFL, are
optimized for network division at the initial layers, which decreases the
protection of the client data and increases communication overhead. In this
paper, we demonstrate that splitting depth is crucial for maintaining privacy
and communication efficiency in distributed training. We also show that MocoSFL
suffers from a catastrophic quality deterioration for the minimal communication
overhead. As a remedy, we introduce Momentum-Aligned contrastive Split
Federated Learning (MonAcoSFL), which aligns online and momentum client models
during training procedure. Consequently, we achieve state-of-the-art accuracy
while significantly reducing the communication overhead, making MonAcoSFL more
practical in real-world scenarios.

通过分割网络层，将网络层分配给客户端设备和中央服务器，我们介绍了一种名为 MonAcoSFL 的 Momentum-Aligned contrastive Split Federated Learning 方法，通过保持深度分割来维护隐私和通信效率，从而在降低通信开销的同时实现了最先进的准确性。

对分割联邦自监督学习进行深入剖析

A deep cut into Split Federated Self-supervised Learning

A major bottleneck in training robust Human-Activity Recognition models (HAR)
is the need for large-scale labeled sensor datasets. Because labeling large
amounts of sensor data is an expensive task, unsupervised and semi-supervised
learning techniques have emerged that can learn good features from the data
without requiring any labels. In this paper, we extend this line of research
and present a novel technique called Collaborative Self-Supervised Learning
(ColloSSL) which leverages unlabeled data collected from multiple devices worn
by a user to learn high-quality features of the data. A key insight that
underpins the design of ColloSSL is that unlabeled sensor datasets
simultaneously captured by multiple devices can be viewed as natural
transformations of each other, and leveraged to generate a supervisory signal
for representation learning. We present three technical innovations to extend
conventional self-supervised learning algorithms to a multi-device setting: a
Device Selection approach which selects positive and negative devices to enable
contrastive learning, a Contrastive Sampling algorithm which samples positive
and negative examples in a multi-device setting, and a loss function called
Multi-view Contrastive Loss which extends standard contrastive loss to a
multi-device setting. Our experimental results on three multi-device datasets
show that ColloSSL outperforms both fully-supervised and semi-supervised
learning techniques in majority of the experiment settings, resulting in an
absolute increase of upto 7.9% in F_1 score compared to the best performing
baselines. We also show that ColloSSL outperforms the fully-supervised methods
in a low-data regime, by just using one-tenth of the available labeled data in
the best case.

本文介绍了一种名为 Collaborative Self-Supervised Learning（ColloSSL）的新技术，利用用户多个设备收集的未标记数据来学习数据的优质特征，实验结果表明，ColloSSL 在多个实验设置中均优于全监督和半监督学习技术，与最佳基准测试成绩相比，F_1 得分的绝对增量最高达 7.9％；此外，ColloSSL 在低数据情况下也表现优异，仅利用最佳情况下可用的十分之一的标记数据即可优于全监督方法。