Personalized Federated Learning (PFL) represents a promising solution for decentralized learning in heterogeneous data environments. Partial model personalization has been proposed to improve the efficiency of PFL by selectively updating local model parameters instead of aggregating all of them. However, previous work on partial model personalization has mainly focused on Convolutional Neural Networks (CNNs), leaving a gap in understanding how it can be applied to other popular models such as Vision Transformers (ViTs). In this work, we investigate where and how to partially personalize a ViT model. Specifically, we empirically evaluate the sensitivity to data distribution of each type of layer. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix, which leverages plugins to transfer information from the aggregated model to the local client as a personalization. Finally, we evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness in improving the model's performance compared to several advanced PFL methods.

个性化联邦学习 (PFL) 是在异构数据环境中进行去中心化学习的一种有前景的解决方案。通过选择性地更新本地模型参数而不是聚合所有参数，局部模型个性化提高了 PFL 的效率。本研究探讨了如何在 Vision Transformers (ViTs) 这类流行模型中部分个性化模型，根据各类型层对数据分布的敏感度，我们提出了一种名为 FedPerfix 的新方法，它利用插件将从聚合模型转移到本地客户端作为个性化信息。最后，我们通过对 CIFAR-100、OrganAMNIST 和 Office-Home 数据集进行评估，并与多种先进的 PFL 方法进行比较，证明了所提方法在提高模型性能方面的有效性。

FedPerfix: 面向联邦学习的视觉Transformer部分模型个性化