This paper investigates the feasibility of federated representation learning
under the constraints of communication cost and privacy protection. Existing
works either conduct annotation-guided local training which requires frequent
communication or aggregates the client models via weight averaging which has
potential risks of privacy exposure. To tackle the above problems, we first
identify that self-supervised contrastive local training is robust against the
non-identically distributed data, which provides the feasibility of longer
local training and thus reduces the communication cost. Then based on the
aforementioned robustness, we propose a novel Federated representation Learning
framework with Ensemble Similarity Distillation~(FLESD) that utilizes this
robustness. At each round of communication, the server first gathers a fraction
of the clients' inferred similarity matrices on a public dataset. Then it
ensembles the similarity matrices and train the global model via similarity
distillation. We verify the effectiveness of FLESD by a series of empirical
experiments and show that, despite stricter constraints, it achieves comparable
results under multiple settings on multiple datasets.

本文探讨了在通信成本和隐私保护的限制下联邦表示学习的可行性。首先证明了自监督对比本地训练的鲁棒性，然后基于此提出了一种新的 Federated representation Learning 框架，使用相似度蒸馏对全局模型进行训练。我们通过一系列实验验证了这种方法的有效性，并表明在多种数据集上，尽管受到更严格的限制，它仍能够以可比较的结果实现多种设置下的性能。