Federated learning (FL) has emerged as a promising paradigm for fine-tuning foundation models using distributed data in a privacy-preserving manner. Under limited computational resources, clients often find it more practical to fine-tune a selected subset of layers, rather than the entire model, based on their task-specific data. In this study, we provide a thorough theoretical exploration of selective layer fine-tuning in FL, emphasizing a flexible approach that allows the clients to adjust their selected layers according to their local data and resources. We theoretically demonstrate that the layer selection strategy has a significant impact on model convergence in two critical aspects: the importance of selected layers and the heterogeneous choices across clients. Drawing from these insights, we further propose a strategic layer selection method that utilizes local gradients and regulates layer selections across clients. The extensive experiments on both image and text datasets demonstrate the effectiveness of the proposed strategy compared with several baselines, highlighting its advances in identifying critical layers that adapt to the client heterogeneity and training dynamics in FL.

本研究针对联邦学习中资源有限的情况下，客户在微调模型时通常仅选择部分层进行调整的问题进行了深入理论探讨。我们提出了一种灵活的方法，允许客户根据本地数据和资源调整选层策略，并证明该策略对模型收敛性有显著影响。我们的实验证明了此方法在识别关键层方面的有效性，特别是在客户异质性和训练动态下的适应性。 

在联邦学习中探索选择性层微调