Federated learning (FL) is a distributed learning paradigm that maximizes the
potential of data-driven models for edge devices without sharing their raw
data. However, devices often have non-independent and identically distributed
(non-IID) data, meaning their local data distributions can vary significantly.
The heterogeneity in input data distributions across devices, commonly referred
to as the feature shift problem, can adversely impact the training convergence
and accuracy of the global model. To analyze the intrinsic causes of the
feature shift problem, we develop a generalization error bound in FL, which
motivates us to propose FedCiR, a client-invariant representation learning
framework that enables clients to extract informative and client-invariant
features. Specifically, we improve the mutual information term between
representations and labels to encourage representations to carry essential
classification knowledge, and diminish the mutual information term between the
client set and representations conditioned on labels to promote representations
of clients to be client-invariant. We further incorporate two regularizers into
the FL framework to bound the mutual information terms with an approximate
global representation distribution to compensate for the absence of the
ground-truth global representation distribution, thus achieving informative and
client-invariant feature extraction. To achieve global representation
distribution approximation, we propose a data-free mechanism performed by the
server without compromising privacy. Extensive experiments demonstrate the
effectiveness of our approach in achieving client-invariant representation
learning and solving the data heterogeneity issue.

提出了一种名为 FedCiR 的客户端不变表示学习框架，通过改进表示和标签之间的互信息项，以及减少客户集和表示之间在标签条件下的互信息项，实现提取信息性和客户端不变的特征，通过两个正则化项来限制近似全局表示分布的互信息项以弥补缺失的真实全局表示分布，进而实现了解决数据异质性和实现客户端不变表示学习的目标。