When personalized federated learning (FL) meets large foundation models, new challenges arise from various limitations in resources. In addition to typical limitations such as data, computation, and communication costs, access to the models is also often limited. This paper endeavors to solve both the challenges of limited resources and personalization. i.e., distribution shifts between clients. To do so, we propose a method named ZOOPFL that uses Zeroth-Order Optimization for Personalized Federated Learning. ZOOPFL avoids direct interference with the foundation models and instead learns to adapt its inputs through zeroth-order optimization. In addition, we employ simple yet effective linear projections to remap its predictions for personalization. To reduce the computation costs and enhance personalization, we propose input surgery to incorporate an auto-encoder with low-dimensional and client-specific embeddings. We provide theoretical support for ZOOPFL to analyze its convergence. Extensive empirical experiments on computer vision and natural language processing tasks using popular foundation models demonstrate its effectiveness for FL on black-box foundation models.

当个性化联邦学习 (FL) 遇到大型基础模型时，面临着资源限制的多种挑战。本文提出了一种名为 ZOOPFL 的方法，使用零阶优化来解决限制资源和个性化之间的挑战，通过简单而有效的线性投影实现预测和个性化，并通过输入手术方法减少计算成本和增强个性化。在流行的基础模型上进行的广泛实验表明，ZOOPFL 对于黑盒基础模型的 FL 具有很好的效果。

探索个性化联邦学习的黑盒基础模型：ZooPFL