Abstract:Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. Nevertheless, FL encounters challenges due to non-independent and identically distributed (non-i.i.d) data, leading to potential performance degradation and hindered convergence. While prior studies predominantly addressed the issue of skewed label distribution, our research addresses a crucial yet frequently overlooked problem known as multi-domain FL. In this scenario, clients' data originate from diverse domains with distinct feature distributions, instead of label distributions. To address the multi-domain problem in FL, we propose a novel method called Federated learning Without normalizations (FedWon). FedWon draws inspiration from the observation that batch normalization (BN) faces challenges in effectively modeling the statistics of multiple domains, while existing normalization techniques possess their own limitations. In order to address these issues, FedWon eliminates the normalization layers in FL and reparameterizes convolution layers with scaled weight standardization. Through extensive experimentation on five datasets and five models, our comprehensive experimental results demonstrate that FedWon surpasses both FedAvg and the current state-of-the-art method (FedBN) across all experimental setups, achieving notable accuracy improvements of more than 10% in certain domains. Furthermore, FedWon is versatile for both cross-silo and cross-device FL, exhibiting robust domain generalization capability, showcasing strong performance even with a batch size as small as 1, thereby catering to resource-constrained devices. Additionally, FedWon can also effectively tackle the challenge of skewed label distribution.

FedBnR: Mitigating federated learning Non-IID problem by breaking the skewed task and reconstructing representation

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

FedWon: Triumphing Multi-domain Federated Learning Without Normalization

Aligning Model Outputs for Class Imbalanced Non-Iid Federated Learning

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction

A Simple Data Augmentation for Feature Distribution Skewed Federated Learning

FedIBD: a federated learning framework in asynchronous mode for imbalanced data

FedUP: Bridging Fairness and Efficiency in Cross-Silo Federated Learning

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

FedNor: A Robust Training Framework for Federated Learning Based on Normal Aggregation

Why Batch Normalization Damage Federated Learning on Non-IID Data?

Exploring Personalization Via Federated Representation Learning on Non-Iid Data.

An Optimization Method for Non-IID Federated Learning Based on Deep Reinforcement Learning

Adaptive Block-Wise Regularization and Knowledge Distillation for Enhancing Federated Learning

Towards Instance-adaptive Inference for Federated Learning

FedBranch: Heterogeneous Federated Learning Via Multi-Branch Neural Network.

Data Augmentation Based Federated Learning

Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains

Batch Normalization Damages Federated Learning on NON-IID Data: Analysis and Remedy