Abstract:Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.

A unified framework for multi-modal federated learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

Multimodal Federated Learning: A Survey

Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection

A survey of multimodal federated learning: background, applications, and perspectives

Balanced Multi-modal Federated Learning via Cross-Modal Infiltration

FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

Multimodal Federated Learning

Multimodal Federated Learning via Contrastive Representation Ensemble

FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning

FedMultimodal: A Benchmark for Multimodal Federated Learning

Federated Mutual Learning: a Collaborative Machine Learning Method for Heterogeneous Data, Models, and Objectives

A Multimodal Biometric Recognition Method Based on Federated Learning

Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation

Multimodal federated learning: Concept, methods, applications and future directions

Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality

3FM: Multi-modal Meta-learning for Federated Tasks

FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data