Many mobile sensing applications utilize data from various modalities,
including motion and physiological sensors in mobile and wearable devices.
Federated Learning (FL) is particularly suitable for these applications thanks
to its privacy-preserving feature. However, challenges such as limited battery
life, poor network conditions, and sensor malfunctions can restrict the use of
all available modalities for local model training. Additionally, existing
multimodal FL systems also struggle with scalability and efficiency as the
number of modality sources increases. To address these issues, we introduce
FLISM, a framework designed to enable multimodal FL with incomplete modalities.
FLISM leverages simulation technique to learn robust representations that can
handle missing modalities and transfers model knowledge across clients with
varying set of modalities. The evaluation results using three real-world
datasets and simulations demonstrate FLISM's effective balance between model
performance and system efficiency. It shows an average improvement of .067 in
F1-score, while also reducing communication (2.69x faster) and computational
(2.28x more efficient) overheads compared to existing methods addressing
incomplete modalities. Moreover, in simulated scenarios involving tasks with a
larger number of modalities, FLISM achieves a significant speedup of
3.23x~85.10x in communication and 3.73x~32.29x in computational efficiency.

通过使用模拟技术，FLISM 框架能够学习稳健的表示，以处理缺失的传感器模态，并在模态不同的客户端之间传递模型知识。通过使用三个真实世界数据集和模拟结果进行评估，FLISM 在模型性能和系统效率之间取得了有效的平衡，提高了 F1-score 的 0.067 平均改进，并相较于解决不完整模态的现有方法，减少了通信（2.69 倍更快）和计算（2.28 倍更高效）开销。此外，在涉及更多模态任务的模拟场景中，FLISM 在通信效率上实现了 3.23 倍～85.10 倍的显著加速和 3.73 倍～32.29 倍的计算效率提升。