Many mobile sensing applications utilize data from various modalities,
including motion and physiological sensors in mobile and wearable devices.
Federated Learning (FL) is particularly suitable for these applications thanks
to its privacy-preserving feature. However, challenges such as limited battery
life, poor network conditions, and sensor malfunctions can restrict the use of
all available modalities for local model training. Additionally, existing
multimodal FL systems also struggle with scalability and efficiency as the
number of modality sources increases. To address these issues, we introduce
FLISM, a framework designed to enable multimodal FL with incomplete modalities.
FLISM leverages simulation technique to learn robust representations that can
handle missing modalities and transfers model knowledge across clients with
varying set of modalities. The evaluation results using three real-world
datasets and simulations demonstrate FLISM's effective balance between model
performance and system efficiency. It shows an average improvement of .067 in
F1-score, while also reducing communication (2.69x faster) and computational
(2.28x more efficient) overheads compared to existing methods addressing
incomplete modalities. Moreover, in simulated scenarios involving tasks with a
larger number of modalities, FLISM achieves a significant speedup of
3.23x~85.10x in communication and 3.73x~32.29x in computational efficiency.

通过使用模拟技术，FLISM 框架能够学习稳健的表示，以处理缺失的传感器模态，并在模态不同的客户端之间传递模型知识。通过使用三个真实世界数据集和模拟结果进行评估，FLISM 在模型性能和系统效率之间取得了有效的平衡，提高了 F1-score 的 0.067 平均改进，并相较于解决不完整模态的现有方法，减少了通信（2.69 倍更快）和计算（2.28 倍更高效）开销。此外，在涉及更多模态任务的模拟场景中，FLISM 在通信效率上实现了 3.23 倍～85.10 倍的显著加速和 3.73 倍～32.29 倍的计算效率提升。

不完整感知模式下的联邦学习

Federated Learning with Incomplete Sensing Modalities

Federated learning (FL) underpins advancements in privacy-preserving
distributed computing by collaboratively training neural networks without
exposing clients' raw data. Current FL paradigms primarily focus on uni-modal
data, while exploiting the knowledge from distributed multimodal data remains
largely unexplored. Existing multimodal FL (MFL) solutions are mainly designed
for statistical or modality heterogeneity from the input side, however, have
yet to solve the fundamental issue,"modality imbalance", in distributed
conditions, which can lead to inadequate information exploitation and
heterogeneous knowledge aggregation on different modalities.In this paper, we
propose a novel Cross-Modal Infiltration Federated Learning (FedCMI) framework
that effectively alleviates modality imbalance and knowledge heterogeneity via
knowledge transfer from the global dominant modality. To avoid the loss of
information in the weak modality due to merely imitating the behavior of
dominant modality, we design the two-projector module to integrate the
knowledge from dominant modality while still promoting the local feature
exploitation of weak modality. In addition, we introduce a class-wise
temperature adaptation scheme to achieve fair performance across different
classes. Extensive experiments over popular datasets are conducted and give us
a gratifying confirmation of the proposed framework for fully exploring the
information of each modality in MFL.

通过知识从全局优势形式中转移，我们提出了一种新颖的跨模态渗透联邦学习（FedCMI）框架，有效缓解分布式条件下的模态失衡和知识异质性问题，并且在弱模态中促进局部特征开发，实现了公平的类别性能表现。