Abstract:With the rapid development of imaging sensor technology in the field of remote sensing, multi-modal remote sensing data fusion has emerged as a crucial research direction for land cover classification tasks. While diffusion models have made great progress in generative models and image classification tasks, existing models primarily focus on single-modality and single-client control, that is, the diffusion process is driven by a single modal in a single computing node. To facilitate the secure fusion of heterogeneous data from clients, it is necessary to enable distributed multi-modal control, such as merging the hyperspectral data of organization A and the LiDAR data of organization B privately on each base station client. In this study, we propose a multi-modal collaborative diffusion federated learning framework called FedDiff. Our framework establishes a dual-branch diffusion model feature extraction setup, where the two modal data are inputted into separate branches of the encoder. Our key insight is that diffusion models driven by different modalities are inherently complementary in terms of potential denoising steps on which bilateral connections can be built. Considering the challenge of private and efficient communication between multiple clients, we embed the diffusion model into the federated learning communication structure, and introduce a lightweight communication module. Qualitative and quantitative experiments validate the superiority of our framework in terms of image quality and conditional consistency.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address two main issues in multimodal remote sensing data fusion: 1. **Limitations in Multimodal Data Feature Extraction**: - Current multimodal remote sensing data fusion methods primarily consider spatial and spectral information during feature extraction, lacking consideration of frequency domain analysis. Frequency domain filtering can enhance or remove specific ground features in remote sensing images by analyzing components at specific frequencies. - Single-modal satellite data (e.g., hyperspectral satellites) often struggle to provide accurate structural information, especially in cases of cloud cover or significant atmospheric interference. LiDAR data can provide complementary data such as surface height information and ground structure details, thereby improving the comprehensiveness and accuracy of remote sensing applications. 2. **Privacy and Communication Efficiency Issues in Distributed Multimodal Data Fusion**: - In multi-satellite distributed systems, certain types of data contain sensitive information, with public access restricted to level 1 or higher-level products, while level 0 data is protected by privacy laws. - Traditional data transmission methods involve transmitting raw data between multiple ground stations, posing a risk of sensitive information being exposed to attackers. This not only affects privacy protection but also raises issues of data integrity and security. - Existing fusion processes mainly rely on fusion and computation on a single server. Due to the limitations of data transmission and storage costs, this method struggles to achieve efficient data processing. To address these issues, the paper proposes a federated learning framework based on a diffusion model (FedDiff), which enables secure and efficient fusion of multimodal data in a distributed environment while ensuring data privacy. Specifically, FedDiff addresses the aforementioned issues through the following points: - **Dual-Branch Diffusion Model**: Establishes a dual-branch diffusion model feature extraction architecture, where different modalities of data are input into different branches of the encoder. Utilizing the complementarity of different modality-driven diffusion models in latent denoising steps, bilateral connections are established. - **Lightweight Communication Module**: Introduces a lightweight communication module to reduce communication costs between multiple clients, ensuring data privacy during transmission. - **Multimodal Federated Learning**: Embeds the diffusion model into the federated learning communication structure to achieve distributed fusion of multimodal data, improving classification performance and communication efficiency. Through these innovations, FedDiff achieves an average classification accuracy of 96.77% on three multimodal datasets while significantly reducing communication costs.

FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients