Federated Learning has become an important learning paradigm due to its
privacy and computational benefits. As the field advances, two key challenges
that still remain to be addressed are: (1) system heterogeneity - variability
in the compute and/or data resources present on each client, and (2) lack of
labeled data in certain federated settings. Several recent developments have
tried to overcome these challenges independently. In this work, we propose a
unified and systematic framework, \emph{Heterogeneous Self-supervised Federated
Learning} (Hetero-SSFL) for enabling self-supervised learning with federation
on heterogeneous clients. The proposed framework allows collaborative
representation learning across all the clients without imposing architectural
constraints or requiring presence of labeled data. The key idea in Hetero-SSFL
is to let each client train its unique self-supervised model and enable the
joint learning across clients by aligning the lower dimensional representations
on a common dataset. The entire training procedure could be viewed as self and
peer-supervised as both the local training and the alignment procedures do not
require presence of any labeled data. As in conventional self-supervised
learning, the obtained client models are task independent and can be used for
varied end-tasks. We provide a convergence guarantee of the proposed framework
for non-convex objectives in heterogeneous settings and also empirically
demonstrate that our proposed approach outperforms the state of the art methods
by a significant margin.

提出了一种名为 Heterogeneous Self-supervised Federated Learning（Hetero-SSFL）的统一框架，可以在异构客户端上进行协作表示学习，同时解决了系统异质性和标记数据匮乏等问题，并在非凸目标的异构性设置中提供了收敛保证，而且比现有方法表现更佳。