Multimodal Federated Learning (MMFL) utilizes multiple modalities in each
client to build a more powerful Federated Learning (FL) model than its unimodal
counterpart. However, the impact of missing modality in different clients, also
called modality incongruity, has been greatly overlooked. This paper, for the
first time, analyses the impact of modality incongruity and reveals its
connection with data heterogeneity across participating clients. We
particularly inspect whether incongruent MMFL with unimodal and multimodal
clients is more beneficial than unimodal FL. Furthermore, we examine three
potential routes of addressing this issue. Firstly, we study the effectiveness
of various self-attention mechanisms towards incongruity-agnostic information
fusion in MMFL. Secondly, we introduce a modality imputation network (MIN)
pre-trained in a multimodal client for modality translation in unimodal clients
and investigate its potential towards mitigating the missing modality problem.
Thirdly, we assess the capability of client-level and server-level
regularization techniques towards mitigating modality incongruity effects.
Experiments are conducted under several MMFL settings on two publicly available
real-world datasets, MIMIC-CXR and Open-I, with Chest X-Ray and radiology
reports.

多模态联邦学习 (MMFL) 利用每个客户端的多个模态构建比单模态对应物更强大的联邦学习 (FL) 模型。然而，不同客户端的模态不一致性对其影响一直被大大忽视。本文首次分析模态不一致性的影响，并揭示其与参与客户端的数据异质性的联系。我们特别研究了不一致的多模态客户端和单模态 FL 的可行性，并研究了解决此问题的三种潜在途径。首先，我们研究了各种自注意机制对 MMFL 中的不一致性不可知信息融合的有效性。其次，我们引入了在多模态客户端中预先训练的模态补全网络 (MIN)，研究其在单模态客户端中模态转换中减轻缺失模态问题的潜力。第三，我们评估了客户端级和服务器级正则化技术对减轻模态不一致性效果的能力。实验证明在两个公开的真实世界数据集 MIMIC-CXR 和 Open-I 上，我们在几种 MMFL 设置下进行了实验，其中包括胸部 X 射线和放射学报告。