Many mobile sensing applications utilize data from various modalities,
including motion and physiological sensors in mobile and wearable devices.
Federated Learning (FL) is particularly suitable for these applications thanks
to its privacy-preserving feature. However, challenges such as limited battery
life, poor network conditions, and sensor malfunctions can restrict the use of
all available modalities for local model training. Additionally, existing
multimodal FL systems also struggle with scalability and efficiency as the
number of modality sources increases. To address these issues, we introduce
FLISM, a framework designed to enable multimodal FL with incomplete modalities.
FLISM leverages simulation technique to learn robust representations that can
handle missing modalities and transfers model knowledge across clients with
varying set of modalities. The evaluation results using three real-world
datasets and simulations demonstrate FLISM's effective balance between model
performance and system efficiency. It shows an average improvement of .067 in
F1-score, while also reducing communication (2.69x faster) and computational
(2.28x more efficient) overheads compared to existing methods addressing
incomplete modalities. Moreover, in simulated scenarios involving tasks with a
larger number of modalities, FLISM achieves a significant speedup of
3.23x~85.10x in communication and 3.73x~32.29x in computational efficiency.

通过使用模拟技术，FLISM 框架能够学习稳健的表示，以处理缺失的传感器模态，并在模态不同的客户端之间传递模型知识。通过使用三个真实世界数据集和模拟结果进行评估，FLISM 在模型性能和系统效率之间取得了有效的平衡，提高了 F1-score 的 0.067 平均改进，并相较于解决不完整模态的现有方法，减少了通信（2.69 倍更快）和计算（2.28 倍更高效）开销。此外，在涉及更多模态任务的模拟场景中，FLISM 在通信效率上实现了 3.23 倍～85.10 倍的显著加速和 3.73 倍～32.29 倍的计算效率提升。

不完整感知模式下的联邦学习

Federated Learning with Incomplete Sensing Modalities

Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is
desirable to joint learning of multimodal images. However, in clinical
practice, it is not always possible to acquire a complete set of MRIs, and the
problem of missing modalities causes severe performance degradation in existing
multimodal segmentation methods. In this work, we present the first attempt to
exploit the Transformer for multimodal brain tumor segmentation that is robust
to any combinatorial subset of available modalities. Concretely, we propose a
novel multimodal Medical Transformer (mmFormer) for incomplete multimodal
learning with three main components: the hybrid modality-specific encoders that
bridge a convolutional encoder and an intra-modal Transformer for both local
and global context modeling within each modality; an inter-modal Transformer to
build and align the long-range correlations across modalities for
modality-invariant features with global semantics corresponding to tumor
region; a decoder that performs a progressive up-sampling and fusion with the
modality-invariant features to generate robust segmentation. Besides, auxiliary
regularizers are introduced in both encoder and decoder to further enhance the
model's robustness to incomplete modalities. We conduct extensive experiments
on the public BraTS $2018$ dataset for brain tumor segmentation. The results
demonstrate that the proposed mmFormer outperforms the state-of-the-art methods
for incomplete multimodal brain tumor segmentation on almost all subsets of
incomplete modalities, especially by an average 19.07% improvement of Dice on
tumor segmentation with only one available modality. The code is available at
this https URL

该研究提出了一种多模态医学 Transformer (mnFormer)，用于不完整多模态学习的脑肿瘤分割，并且该方法在 BraTS 2018 数据集上显示出了比现有方法更好的表现。