FedMMR: Multi-Modal Federated Learning Via Missing Modality Reconstruction

Shu Wang,Zhe Qu,Yuan Liu,Shichao Kan,Yixiong Liang,Jianxin Wang
DOI: https://doi.org/10.1109/icme57554.2024.10688162
2024-01-01
Abstract:Federated Learning (FL) presents a decentralized learning approach for privacy preservation. While many research focuses on uni-modal FL, a more intricate version, multimodal FL, uncovers fundamental attributes. Clients might only gather specific modalities, incurring missing modalities in multi-modal FL. The missing modality problem incurs modality heterogeneity among clients and loses inter-modal connection within one client. To address these issues, we introduce a novel multi-modal FL method called Federated Missing Modality Reconstruction (FedMMR), which tackles these dual challenges through two distinct facets. First, we devise a cross-modal reconstruction policy for synthesizing absent modalities from other modalities. This aligns the data feature spaces across clients, thereby alleviating bias in resultant local models. Subsequently, we steer local models to maintain an inter-modal awareness of both existing and reconstructed modalities, recognizing their potential as complementary components. Comprehensive results show that FedMMR outperforms existing FL baselines.
What problem does this paper attempt to address?