With the rapid development of imaging sensor technology in the field of
remote sensing, multi-modal remote sensing data fusion has emerged as a crucial
research direction for land cover classification tasks. While diffusion models
have made great progress in generative models and image classification tasks,
existing models primarily focus on single-modality and single-client control,
that is, the diffusion process is driven by a single modal in a single
computing node. To facilitate the secure fusion of heterogeneous data from
clients, it is necessary to enable distributed multi-modal control, such as
merging the hyperspectral data of organization A and the LiDAR data of
organization B privately on each base station client. In this study, we propose
a multi-modal collaborative diffusion federated learning framework called
FedDiff. Our framework establishes a dual-branch diffusion model feature
extraction setup, where the two modal data are inputted into separate branches
of the encoder. Our key insight is that diffusion models driven by different
modalities are inherently complementary in terms of potential denoising steps
on which bilateral connections can be built. Considering the challenge of
private and efficient communication between multiple clients, we embed the
diffusion model into the federated learning communication structure, and
introduce a lightweight communication module. Qualitative and quantitative
experiments validate the superiority of our framework in terms of image quality
and conditional consistency.

本研究提出了一种名为 FedDiff 的多模态协同扩散联邦学习框架，旨在解决多模态遥感数据融合中的安全和效率通信问题，并通过定性和定量实验证实了该框架在图像质量和条件一致性方面的超越优势。