Federated learning (FL) has obtained tremendous progress in providing
collaborative training solutions for distributed data silos with privacy
guarantees. However, few existing works explore a more realistic scenario where
the clients hold multiple data modalities. In this paper, we aim to solve a
novel challenge in multi-modal federated learning (MFL) -- modality missing --
the clients may lose part of the modalities in their local data sets. To tackle
the problems, we propose a novel multi-modal federated learning method,
Federated Multi-modal contrastiVe training with Pre-trained completion
(FedMVP), which integrates the large-scale pre-trained models to enhance the
federated training. In the proposed FedMVP framework, each client deploys a
large-scale pre-trained model with frozen parameters for modality completion
and representation knowledge transfer, enabling efficient and robust local
training. On the server side, we utilize generated data to uniformly measure
the representation similarity among the uploaded client models and construct a
graph perspective to aggregate them according to their importance in the
system. We demonstrate that the model achieves superior performance over two
real-world image-text classification datasets and is robust to the performance
degradation caused by missing modality.

本文提出了一种新颖的多模态联邦学习方法 FedMVP，通过集成大规模预训练模型来增强联邦训练，解决了多模态数据集中可能丢失模态的问题，在客户端和服务器端分别采用不同的方法进行模态补全和模型聚合，并在实际图像 - 文本分类数据集上证明了该方法的卓越性能和对缺失模态的鲁棒性。