Recently, foundation models, particularly large language models (LLMs), have demonstrated an impressive ability to adapt to various tasks by fine-tuning large amounts of instruction data. Notably, federated foundation models emerge as a privacy preservation method to fine-tune models collaboratively under federated learning (FL) settings by leveraging many distributed datasets with non-IID data. To alleviate communication and computation overhead, parameter-efficient methods are introduced for efficiency, and some research adapted personalization methods to federated foundation models for better user preferences alignment. However, a critical gap in existing research is the neglect of test-time distribution shifts in real-world applications. Therefore, to bridge this gap, we propose a new setting, termed test-time personalization, which not only concentrates on the targeted local task but also extends to other tasks that exhibit test-time distribution shifts. To address challenges in this new setting, we explore a simple yet effective solution to learn a comprehensive foundation model. Specifically, a dual-personalizing adapter architecture (FedDPA) is proposed, comprising a global adapter and a local adapter for addressing test-time distribution shifts and personalization, respectively. Additionally, we introduce an instance-wise dynamic weighting mechanism to optimize the balance between the global and local adapters, enhancing overall performance. The effectiveness of the proposed method has been evaluated on benchmark datasets across different NLP tasks.

提出了一个新的设置，称为测试时个性化，其中不仅专注于目标的本地任务，而且扩展到展示测试时分布偏移的其他任务。为了解决这个新设置中的挑战，提出了一个名为FedDPA的双向个性化适配器架构，包括全局适配器和局部适配器，用于解决测试时分布偏移和个性化问题，并引入了一个实例动态加权机制来优化全局和局部适配器之间的平衡，提高整体性能。在不同的自然语言处理任务上对所提出的方法进行了评估。

用于联邦基础模型的双个性化适配器