Abstract:Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.

FedSea: Federated Learning Via Selective Feature Alignment for Non-IID Multimodal Data

FedSiam-DA: Dual-aggregated Federated Learning Via Siamese Network for Non-Iid Data

FedEL: Federated ensemble learning for non-iid data

A Survey of Federated Learning on Non-IID Data

Feature Matching Data Synthesis for Non-IID Federated Learning

FedAA: Using Non-sensitive Modalities to Improve Federated Learning while Preserving Image Privacy

Federated Learning for Non-IID Data Via Unified Feature Learning and Optimization Objective Alignment

FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

FedMultimodal: A Benchmark for Multimodal Federated Learning

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning

FedMCSA: Personalized federated learning via model components self-attention

FedDA: Resource-adaptive Federated Learning with Dual-Alignment Aggregation Optimization for Heterogeneous Edge Devices

FedSeq: A Hybrid Federated Learning Framework Based on Sequential In-Cluster Training.

Enhanced Federated Learning on Non-Iid Data Via Local Importance Sampling

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

Federated learning on non-IID and long-tailed data via dual-decoupling

Federated Loss Exploration for Improved Convergence on Non-IID Data