We investigate contrastive learning in the federated setting through the lens
of SimCLR and multi-view mutual information maximization. In doing so, we
uncover a connection between contrastive representation learning and user
verification; by adding a user verification loss to each client's local SimCLR
loss we recover a lower bound to the global multi-view mutual information. To
accommodate for the case of when some labelled data are available at the
clients, we extend our SimCLR variant to the federated semi-supervised setting.
We see that a supervised SimCLR objective can be obtained with two changes: a)
the contrastive loss is computed between datapoints that share the same label
and b) we require an additional auxiliary head that predicts the correct labels
from either of the two views. Along with the proposed SimCLR extensions, we
also study how different sources of non-i.i.d.-ness can impact the performance
of federated unsupervised learning through global mutual information
maximization; we find that a global objective is beneficial for some sources of
non-i.i.d.-ness but can be detrimental for others. We empirically evaluate our
proposed extensions in various tasks to validate our claims and furthermore
demonstrate that our proposed modifications generalize to other pretraining
methods.

我们通过 SimCLR 和多视图互信息最大化的角度研究了联邦环境中的对比学习，其中发现了对比表示学习与用户验证之间的关联；通过在每个客户端的本地 SimCLR loss 加上用户验证损失，我们恢复了全局多视图互信息的下界。为了适应在客户端存在一些标记数据的情况，我们将 SimCLR 变体扩展到联邦半监督设置。我们发现可以通过两个改变来获得受监督的 SimCLR 目标：a) 对比损失是在具有相同标签的数据点之间计算的，b) 我们需要一个额外的辅助头部，从任一视图中预测正确的标签。除了我们提出的 SimCLR 扩展，我们还研究了不同非独立同分布性的来源对联邦无监督学习通过全局互信息最大化的性能的影响；我们发现全局目标对于某些非独立同分布性的来源是有益的，但对于其他来源可能是有害的。我们在各种任务中对我们提出的扩展进行了实证评估，以验证我们的论点，并进一步证明我们的修改方法推广到其他预训练方法。

联邦对比学习的互信息视角

A Mutual Information Perspective on Federated Contrastive Learning

Unsupervised representation learning has achieved outstanding performances
using centralized data available on the Internet. However, the increasing
awareness of privacy protection limits sharing of decentralized unlabeled image
data that grows explosively in multiple parties (e.g., mobile phones and
cameras). As such, a natural problem is how to leverage these data to learn
visual representations for downstream tasks while preserving data privacy. To
address this problem, we propose a novel federated unsupervised learning
framework, FedU. In this framework, each party trains models from unlabeled
data independently using contrastive learning with an online network and a
target network. Then, a central server aggregates trained models and updates
clients' models with the aggregated model. It preserves data privacy as each
party only has access to its raw data. Decentralized data among multiple
parties are normally non-independent and identically distributed (non-IID),
leading to performance degradation. To tackle this challenge, we propose two
simple but effective methods: 1) We design the communication protocol to upload
only the encoders of online networks for server aggregation and update them
with the aggregated encoder; 2) We introduce a new module to dynamically decide
how to update predictors based on the divergence caused by non-IID. The
predictor is the other component of the online network. Extensive experiments
and ablations demonstrate the effectiveness and significance of FedU. It
outperforms training with only one party by over 5% and other methods by over
14% in linear and semi-supervised evaluation on non-IID data.

该研究提出了一种名为 FedU 的联邦无监督学习框架，可以在保护数据隐私的同时利用去中心化的未标记图像数据学习视觉表示，并针对数据的非独立同分布问题提出了有效方法，实验证明其优于其他方法。