Federated learning encounters substantial challenges with heterogeneous data,
leading to performance degradation and convergence issues. While considerable
progress has been achieved in mitigating such an impact, the reliability aspect
of federated models has been largely disregarded. In this study, we conduct
extensive experiments to investigate the reliability of both generic and
personalized federated models. Our exploration uncovers a significant finding:
\textbf{federated models exhibit unreliability when faced with heterogeneous
data}, demonstrating poor calibration on in-distribution test data and low
uncertainty levels on out-of-distribution data. This unreliability is primarily
attributed to the presence of biased projection heads, which introduce
miscalibration into the federated models. Inspired by this observation, we
propose the "Assembled Projection Heads" (APH) method for enhancing the
reliability of federated models. By treating the existing projection head
parameters as priors, APH randomly samples multiple initialized parameters of
projection heads from the prior and further performs targeted fine-tuning on
locally available data under varying learning rates. Such a head ensemble
introduces parameter diversity into the deterministic model, eliminating the
bias and producing reliable predictions via head averaging. We evaluate the
effectiveness of the proposed APH method across three prominent federated
benchmarks. Experimental results validate the efficacy of APH in model
calibration and uncertainty estimation. Notably, APH can be seamlessly
integrated into various federated approaches but only requires less than 30\%
additional computation cost for 100$\times$ inferences within large models.

研究发现异构数据导致联邦学习模型不可靠，提出了一种增强模型可靠性的方法，利用集成投影头消除偏差并提供可靠的预测。

警惕：组装投影头以提高联邦模型的可靠性

Watch Your Head: Assembling Projection Heads to Save the Reliability of  Federated Models

We investigate the role of projection heads, also known as projectors, within
the encoder-projector framework (e.g., SimCLR) used in contrastive learning. We
aim to demystify the observed phenomenon where representations learned before
projectors outperform those learned after -- measured using the downstream
linear classification accuracy, even when the projectors themselves are linear.
In this paper, we make two significant contributions towards this aim.
Firstly, through empirical and theoretical analysis, we identify two crucial
effects -- expansion and shrinkage -- induced by the contrastive loss on the
projectors. In essence, contrastive loss either expands or shrinks the signal
direction in the representations learned by an encoder, depending on factors
such as the augmentation strength, the temperature used in contrastive loss,
etc. Secondly, drawing inspiration from the expansion and shrinkage phenomenon,
we propose a family of linear transformations to accurately model the
projector's behavior. This enables us to precisely characterize the downstream
linear classification accuracy in the high-dimensional asymptotic limit. Our
findings reveal that linear projectors operating in the shrinkage (or
expansion) regime hinder (or improve) the downstream classification accuracy.
This provides the first theoretical explanation as to why (linear) projectors
impact the downstream performance of learned representations. Our theoretical
findings are further corroborated by extensive experiments on both synthetic
data and real image data.

本篇论文旨在揭示对比学习过程中 projection heads 作用的原理，通过实验和理论分析认定对比损失产生的膨胀和收缩效应会影响线性分类的准确性，因此 linears projectors 操作在缩小或者扩张的情况下会妨碍或是提高下游分类的准确性。