When personalized federated learning (FL) meets large foundation models, new
challenges arise from various limitations in resources. In addition to typical
limitations such as data, computation, and communication costs, access to the
models is also often limited. This paper endeavors to solve both the challenges
of limited resources and personalization. i.e., distribution shifts between
clients. To do so, we propose a method named ZOOPFL that uses Zeroth-Order
Optimization for Personalized Federated Learning. ZOOPFL avoids direct
interference with the foundation models and instead learns to adapt its inputs
through zeroth-order optimization. In addition, we employ simple yet effective
linear projections to remap its predictions for personalization. To reduce the
computation costs and enhance personalization, we propose input surgery to
incorporate an auto-encoder with low-dimensional and client-specific
embeddings. We provide theoretical support for ZOOPFL to analyze its
convergence. Extensive empirical experiments on computer vision and natural
language processing tasks using popular foundation models demonstrate its
effectiveness for FL on black-box foundation models.

当个性化联邦学习 (FL) 遇到大型基础模型时，面临着资源限制的多种挑战。本文提出了一种名为 ZOOPFL 的方法，使用零阶优化来解决限制资源和个性化之间的挑战，通过简单而有效的线性投影实现预测和个性化，并通过输入手术方法减少计算成本和增强个性化。在流行的基础模型上进行的广泛实验表明，ZOOPFL 对于黑盒基础模型的 FL 具有很好的效果。