Deep neural networks have shown the ability to extract universal feature
representations from data such as images and text that have been useful for a
variety of learning tasks. However, the fruits of representation learning have
yet to be fully-realized in federated settings. Although data in federated
settings is often non-i.i.d. across clients, the success of centralized deep
learning suggests that data often shares a global feature representation, while
the statistical heterogeneity across clients or tasks is concentrated in the
labels. Based on this intuition, we propose a novel federated learning
framework and algorithm for learning a shared data representation across
clients and unique local heads for each client. Our algorithm harnesses the
distributed computational power across clients to perform many local-updates
with respect to the low-dimensional local parameters for every update of the
representation. We prove that this method obtains linear convergence to the
ground-truth representation with near-optimal sample complexity in a linear
setting, demonstrating that it can efficiently reduce the problem dimension for
each client. This result is of interest beyond federated learning to a broad
class of problems in which we aim to learn a shared low-dimensional
representation among data distributions, for example in meta-learning and
multi-task learning. Further, extensive experimental results show the empirical
improvement of our method over alternative personalized federated learning
approaches in federated environments with heterogeneous data.

本文提出了一个新的联邦学习框架和算法，用于在客户之间学习共享的数据表示和每个客户的独特的本地头。我们的算法利用了跨客户的分布式计算能力，以便每次更新表示时进行许多针对低维本地参数的局部更新。我们证明该方法在线性设置中获得与地面实况表示的线性收敛，并获得接近最优样本复杂度，从而在每个客户端高效地降低问题维度。这个结果超出了联合学习，是一类希望在数据分布中学习共享的低维表示的问题的广泛兴趣，例如在元学习和多任务学习中。此外，广泛的实验结果显示了我们的方法在异构数据的联邦环境中相对于可选的个性化联邦学习方法的经验改进。