Multi-modal transformers mark significant progress in different domains, but
siloed high-quality data hinders their further improvement. To remedy this,
federated learning (FL) has emerged as a promising privacy-preserving paradigm
for training models without direct access to the raw data held by different
clients. Despite its potential, a considerable research direction regarding the
unpaired uni-modal clients and the transformer architecture in FL remains
unexplored. To fill this gap, this paper explores a transfer multi-modal
federated learning (MFL) scenario within the vision-language domain, where
clients possess data of various modalities distributed across different
datasets. We systematically evaluate the performance of existing methods when a
transformer architecture is utilized and introduce a novel framework called
Federated modality complementary and collaboration (FedCola) by addressing the
in-modality and cross-modality gaps among clients. Through extensive
experiments across various FL settings, FedCola demonstrates superior
performance over previous approaches, offering new perspectives on future
federated training of multi-modal transformers.

在视觉语言领域中，通过利用转换器架构对现有方法进行系统评估，并引入一种名为 FedCola 的新框架，填补了关于不配对的单模客户端和 FL 中转换器架构的研究空白。通过在各种 FL 设置下进行广泛实验，FedCola 展示了优于先前方法的性能，为未来多模态转换器的联邦训练提供了新的观点。