In the rapidly evolving field of artificial intelligence, multimodal models, e.g., integrating vision and language into visual-language models (VLMs), have become pivotal for many applications, ranging from image captioning to multimodal search engines. Among these models, the Contrastive Language-Image Pre-training (CLIP) model has demonstrated remarkable performance in understanding and generating nuanced relationships between text and images. However, the conventional training of such models often requires centralized aggregation of vast datasets, posing significant privacy and data governance challenges. To address these concerns, this paper proposes a novel approach that leverages Federated Learning and parameter-efficient adapters, i.e., Low-Rank Adaptation (LoRA), to train VLMs. This methodology preserves data privacy by training models across decentralized data sources and ensures model adaptability and efficiency through LoRA's parameter-efficient fine-tuning. Our approach accelerates training time by up to 34.72 times and requires 2.47 times less memory usage than full fine-tuning.

该论文提出了一种融合Federated Learning和parameter-efficient adapters (LoRA)的方法，以训练视觉语言模型(VLMs)，该方法通过在分散的数据源上训练模型来保护数据隐私，并通过LoRA的参数高效微调来确保模型的适应性和效率。该方法加快了训练速度，比全面微调节省了2.47倍的内存使用。

FLoRA：利用参数高效的联邦学习增强视觉语言模型