Personalized Federated Learning (pFL) has emerged as a promising solution to
tackle data heterogeneity across clients in FL. However, existing pFL methods
either (1) introduce high communication and computation costs or (2) overfit to
local data, which can be limited in scope, and are vulnerable to evolved test
samples with natural shifts. In this paper, we propose PerAda, a
parameter-efficient pFL framework that reduces communication and computational
costs and exhibits superior generalization performance, especially under
test-time distribution shifts. PerAda reduces the costs by leveraging the power
of pretrained models and only updates and communicates a small number of
additional parameters from adapters. PerAda has good generalization since it
regularizes each client's personalized adapter with a global adapter, while the
global adapter uses knowledge distillation to aggregate generalized information
from all clients. Theoretically, we provide generalization bounds to explain
why PerAda improves generalization, and we prove its convergence to stationary
points under non-convex settings. Empirically, PerAda demonstrates competitive
personalized performance (+4.85% on CheXpert) and enables better
out-of-distribution generalization (+5.23% on CIFAR-10-C) on different datasets
across natural and medical domains compared with baselines, while only updating
12.6% of parameters per model based on the adapter.

本文提出了一种名为 PerAda 的参数高效的 Federated Learning 框架，采用预训练模型，使用少量适配器参数进行通信，实现了与基线方法相比更好的个性化性能和广义性能，尤其是在测试时分布发生变化的情况下。