Federated representation learning (FRL) is a popular personalized federated
learning (FL) framework where clients work together to train a common
representation while retaining their personalized heads. Existing studies,
however, largely focus on the over-parameterized regime. In this paper, we make
the initial efforts to investigate FRL in the under-parameterized regime, where
the FL model is insufficient to express the variations in all ground-truth
models. We propose a novel FRL algorithm FLUTE, and theoretically characterize
its sample complexity and convergence rate for linear models in the
under-parameterized regime. To the best of our knowledge, this is the first FRL
algorithm with provable performance guarantees in this regime. FLUTE features a
data-independent random initialization and a carefully designed objective
function that aids the distillation of subspace spanned by the global optimal
representation from the misaligned local representations. On the technical
side, we bridge low-rank matrix approximation techniques with the FL analysis,
which may be of broad interest. We also extend FLUTE beyond linear
representations. Experimental results demonstrate that FLUTE outperforms
state-of-the-art FRL solutions in both synthetic and real-world tasks.

在参数不足的情况下，我们提出了一种新颖的 Federated representation learning 算法 FLUTE，理论上表征了其在线性模型的样本复杂度和收敛速率，实验结果证明其优于现有解决方案。

欠参数化模型中的联邦表示学习

Federated Representation Learning in the Under-Parameterized Regime

This paper investigates the feasibility of federated representation learning
under the constraints of communication cost and privacy protection. Existing
works either conduct annotation-guided local training which requires frequent
communication or aggregates the client models via weight averaging which has
potential risks of privacy exposure. To tackle the above problems, we first
identify that self-supervised contrastive local training is robust against the
non-identically distributed data, which provides the feasibility of longer
local training and thus reduces the communication cost. Then based on the
aforementioned robustness, we propose a novel Federated representation Learning
framework with Ensemble Similarity Distillation~(FLESD) that utilizes this
robustness. At each round of communication, the server first gathers a fraction
of the clients' inferred similarity matrices on a public dataset. Then it
ensembles the similarity matrices and train the global model via similarity
distillation. We verify the effectiveness of FLESD by a series of empirical
experiments and show that, despite stricter constraints, it achieves comparable
results under multiple settings on multiple datasets.

本文探讨了在通信成本和隐私保护的限制下联邦表示学习的可行性。首先证明了自监督对比本地训练的鲁棒性，然后基于此提出了一种新的 Federated representation Learning 框架，使用相似度蒸馏对全局模型进行训练。我们通过一系列实验验证了这种方法的有效性，并表明在多种数据集上，尽管受到更严格的限制，它仍能够以可比较的结果实现多种设置下的性能。