Over the past few years, Federated Learning (FL) has become an emerging
machine learning technique to tackle data privacy challenges through
collaborative training. In the Federated Learning algorithm, the clients submit
a locally trained model, and the server aggregates these parameters until
convergence. Despite significant efforts that have been made to FL in fields
like computer vision, audio, and natural language processing, the FL
applications utilizing multimodal data streams remain largely unexplored. It is
known that multimodal learning has broad real-world applications in emotion
recognition, healthcare, multimedia, and social media, while user privacy
persists as a critical concern. Specifically, there are no existing FL
benchmarks targeting multimodal applications or related tasks. In order to
facilitate the research in multimodal FL, we introduce FedMultimodal, the first
FL benchmark for multimodal learning covering five representative multimodal
applications from ten commonly used datasets with a total of eight unique
modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end
modeling framework ranging from data partition and feature extraction to FL
benchmark algorithms and model evaluation. Unlike existing FL benchmarks,
FedMultimodal provides a standardized approach to assess the robustness of FL
against three common data corruptions in real-life multimodal applications:
missing modalities, missing labels, and erroneous labels. We hope that
FedMultimodal can accelerate numerous future research directions, including
designing multimodal FL algorithms toward extreme data heterogeneity,
robustness multimodal FL, and efficient multimodal FL. The datasets and
benchmark results can be accessed at:
this https URL

引入 FedMultimodal，它是第一个 FL 基准，用于多模态学习，覆盖了来自十个常用数据集的五个代表性多模态应用程序，总共有八种独特方式。与现有的 FL 基准不同，FedMultimodal 提供了一种标准化方法来评估 FL 对实际多模态应用程序中的三种常见数据污染的鲁棒性：缺失模态，缺失标签和错误标签。