The integration of Foundation Models (FMs) with Federated Learning (FL)
presents a transformative paradigm in Artificial Intelligence (AI), offering
enhanced capabilities while addressing concerns of privacy, data
decentralization, and computational efficiency. This paper provides a
comprehensive survey of the emerging field of Federated Foundation Models
(FedFM), elucidating their synergistic relationship and exploring novel
methodologies, challenges, and future directions that the FL research field
needs to focus on in order to thrive in the age of foundation models. A
systematic multi-tiered taxonomy is proposed, categorizing existing FedFM
approaches for model training, aggregation, trustworthiness, and
incentivization. Key challenges, including how to enable FL to deal with high
complexity of computational demands, privacy considerations, contribution
evaluation, and communication efficiency, are thoroughly discussed. Moreover,
the paper explores the intricate challenges of communication, scalability and
security inherent in training/fine-tuning FMs via FL, highlighting the
potential of quantum computing to revolutionize the training, inference,
optimization and data encryption processes. This survey underscores the
importance of further research to propel innovation in FedFM, emphasizing the
need for developing trustworthy solutions. It serves as a foundational guide
for researchers and practitioners interested in contributing to this
interdisciplinary and rapidly advancing field.

综合调查了联邦基础模型（FedFM）领域，阐明了其与联邦学习（FL）的协同关系，并探索了 FL 研究领域在基础模型时代需要关注的新方法、挑战和未来方向。提出了一种系统的多层次分类法，对现有的 FedFM 方法进行了分类。全面讨论了关键挑战，包括如何使 FL 处理高复杂性的计算需求、隐私考虑、贡献评估和通信效率。这项调查强调了在 FedFM 领域进行进一步研究以推动创新的重要性，强调了开发可信解决方案的需求。作为研究人员和从业者参与这个跨学科和快速发展领域的基础指南。

基于基础模型的联邦学习的进展与开放挑战

Advances and Open Challenges in Federated Learning with Foundation  Models

Recently, foundation models, particularly large language models (LLMs), have
demonstrated an impressive ability to adapt to various tasks by fine-tuning
large amounts of instruction data. Notably, federated foundation models emerge
as a privacy preservation method to fine-tune models collaboratively under
federated learning (FL) settings by leveraging many distributed datasets with
non-IID data. To alleviate communication and computation overhead,
parameter-efficient methods are introduced for efficiency, and some research
adapted personalization methods to federated foundation models for better user
preferences alignment. However, a critical gap in existing research is the
neglect of test-time distribution shifts in real-world applications. Therefore,
to bridge this gap, we propose a new setting, termed test-time personalization,
which not only concentrates on the targeted local task but also extends to
other tasks that exhibit test-time distribution shifts. To address challenges
in this new setting, we explore a simple yet effective solution to learn a
comprehensive foundation model. Specifically, a dual-personalizing adapter
architecture (FedDPA) is proposed, comprising a global adapter and a local
adapter for addressing test-time distribution shifts and personalization,
respectively. Additionally, we introduce an instance-wise dynamic weighting
mechanism to optimize the balance between the global and local adapters,
enhancing overall performance. The effectiveness of the proposed method has
been evaluated on benchmark datasets across different NLP tasks.

提出了一个新的设置，称为测试时个性化，其中不仅专注于目标的本地任务，而且扩展到展示测试时分布偏移的其他任务。为了解决这个新设置中的挑战，提出了一个名为 FedDPA 的双向个性化适配器架构，包括全局适配器和局部适配器，用于解决测试时分布偏移和个性化问题，并引入了一个实例动态加权机制来优化全局和局部适配器之间的平衡，提高整体性能。在不同的自然语言处理任务上对所提出的方法进行了评估。

用于联邦基础模型的双个性化适配器

Dual-Personalizing Adapter for Federated Foundation Models

Foundation Models (FMs), such as BERT, GPT, ViT, and CLIP, have demonstrated
remarkable success in a wide range of applications, driven by their ability to
leverage vast amounts of data for pre-training. However, optimizing FMs often
requires access to sensitive data, raising privacy concerns and limiting their
applicability in certain domains. In this paper, we introduce the concept of
Federated Foundation Models (FFMs), a novel approach that combines the benefits
of FMs and Federated Learning (FL) to enable privacy-preserving and
collaborative learning across multiple institutions. We discuss the potential
benefits and challenges of integrating FL into the lifespan of FMs, covering
pre-training, fine-tuning, and application. We further provide formal
definitions of FFM tasks, including FFM pre-training, FFM fine-tuning, and
federated prompt engineering, allowing for more personalized and context-aware
models while maintaining data privacy. Moreover, we explore the possibility of
continual/lifelong learning in FFMs, as increased computational power at the
edge unlocks the potential for optimizing FMs using newly generated private
data at edges. We present experiments and evaluations comparing the performance
of FFMs to traditional FMs on various downstream tasks, demonstrating the
effectiveness of our approach in preserving privacy, reducing overfitting, and
improving model generalizability. The proposed Federated Foundation Models
offer a flexible and scalable framework for training large language models in a
privacy-preserving manner, paving the way for future advancements in both FM
pre-training and federated learning.

本文介绍了一种结合了联邦学习和基础模型的隐私保存方法，即联邦基础模型，其中涉及该方法的不同方面和任务，以及与传统基础模型的比较和实验。