Multimodal federated learning (MFL) has emerged as a decentralized machine
learning paradigm, allowing multiple clients with different modalities to
collaborate on training a machine learning model across diverse data sources
without sharing their private data. However, challenges, such as data
heterogeneity and severely missing modalities, pose crucial hindrances to the
robustness of MFL, significantly impacting the performance of global model. The
absence of a modality introduces misalignment during the local training phase,
stemming from zero-filling in the case of clients with missing modalities.
Consequently, achieving robust generalization in global model becomes
imperative, especially when dealing with clients that have incomplete data. In
this paper, we propose Multimodal Federated Cross Prototype Learning (MFCPL), a
novel approach for MFL under severely missing modalities by conducting the
complete prototypes to provide diverse modality knowledge in modality-shared
level with the cross-modal regularization and modality-specific level with
cross-modal contrastive mechanism. Additionally, our approach introduces the
cross-modal alignment to provide regularization for modality-specific features,
thereby enhancing overall performance, particularly in scenarios involving
severely missing modalities. Through extensive experiments on three multimodal
datasets, we demonstrate the effectiveness of MFCPL in mitigating these
challenges and improving the overall performance.

本文提出了一种名为 “Multimodal Federated Cross Prototype Learning (MFCPL)” 的新方法，针对数据中存在严重缺失模态的多模态联邦学习（MFL）问题，通过使用完整的原型样本在共享模态级别和特定模态级别上运用交叉模态正则化和交叉模态对比机制，为提供多样化的模态知识，引入交叉模态对齐来增强特定模态特征的整体性能，尤其在存在严重缺失模态的场景中表现出优异的效果。通过对三个多模态数据集的广泛实验，我们证明了 MFCPL 在减轻这些挑战和提高整体性能方面的有效性。

基于跨模态原型的严重缺失模态下的多模态联邦学习

Cross-Modal Prototype based Multimodal Federated Learning under Severely  Missing Modality

How to achieve better end-to-end speech translation (ST) by leveraging (text)
machine translation (MT) data? Among various existing techniques, multi-task
learning is one of the effective ways to share knowledge between ST and MT in
which additional MT data can help to learn source-to-target mapping. However,
due to the differences between speech and text, there is always a gap between
ST and MT. In this paper, we first aim to understand this modality gap from the
target-side representation differences, and link the modality gap to another
well-known problem in neural machine translation: exposure bias. We find that
the modality gap is relatively small during training except for some difficult
cases, but keeps increasing during inference due to the cascading effect. To
address these problems, we propose the Cross-modal Regularization with
Scheduled Sampling (Cress) method. Specifically, we regularize the output
predictions of ST and MT, whose target-side contexts are derived by sampling
between ground truth words and self-generated words with a varying probability.
Furthermore, we introduce token-level adaptive training which assigns different
training weights to target tokens to handle difficult cases with large modality
gaps. Experiments and analysis show that our approach effectively bridges the
modality gap, and achieves promising results in all eight directions of the
MuST-C dataset.

本文研究对于 ST 和 MT 之间的通道差，提出了一种名为 Cress 的跨模态规范化方法。通过输出 ST 和 MT 的预测值，使用采样和不同训练权重来处理难以处理的情况，实现了在 MuST-C 数据集的八个方向上取得了显著效果。