Abstract:Combining different data modalities enables deep neural networks to tackle complex tasks more effectively, making multimodal learning increasingly popular. To harness multimodal data closer to end users, it is essential to integrate multimodal learning with privacy-preserving approaches like federated learning (FL). However, compared to conventional unimodal learning, multimodal setting requires dedicated encoders for each modality, resulting in larger and more complex models. Training these models requires significant resources, presenting a substantial challenge for FL clients operating with limited computation and communication resources. To address these challenges, we introduce LW-FedMML, a layer-wise federated multimodal learning approach which decomposes the training process into multiple stages. Each stage focuses on training only a portion of the model, thereby significantly reducing the memory and computational requirements. Moreover, FL clients only need to exchange the trained model portion with the central server, lowering the resulting communication cost. We conduct extensive experiments across various FL and multimodal learning settings to validate the effectiveness of our proposed method. The results demonstrate that LW-FedMML can compete with conventional end-to-end federated multimodal learning (FedMML) while significantly reducing the resource burden on FL clients. Specifically, LW-FedMML reduces memory usage by up to $2.7\times$, computational operations (FLOPs) by $2.4\times$, and total communication cost by $2.3\times$. We also explore a progressive training approach called Prog-FedMML. While it offers lesser resource efficiency than LW-FedMML, Prog-FedMML has the potential to surpass the performance of end-to-end FedMML, making it a viable option for scenarios with fewer resource constraints.

Balanced Multi-modal Federated Learning via Cross-Modal Infiltration

Multimodal Federated Learning: A Survey

A unified framework for multi-modal federated learning

Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection

FedMMR: Multi-Modal Federated Learning Via Missing Modality Reconstruction

Multi-Dimensional Fair Federated Learning

Multimodal Federated Learning via Contrastive Representation Ensemble

Federated Learning on Multimodal Data: A Comprehensive Survey

Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation

Multimodal Federated Learning

FedMultimodal: A Benchmark for Multimodal Federated Learning

Multimodal federated learning: Concept, methods, applications and future directions

A survey of multimodal federated learning: background, applications, and perspectives

FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

Resource-Efficient Federated Multimodal Learning via Layer-wise and Progressive Training

FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

Multimodal Fusion with Block Term Decomposition for Asynchronous Federated Learning

FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning

Prioritizing Modalities: Flexible Importance Scheduling in Federated Multimodal Learning