Abstract:Mobile devices, including smartphones and laptops, generate decentralized and heterogeneous data, presenting significant challenges for traditional centralized machine learning models due to substantial communication costs and privacy risks. Federated Learning (FL) offers a promising alternative by enabling collaborative training of a global model across decentralized devices without data sharing. However, FL faces challenges due to statistical heterogeneity among clients, where non-independent and identically distributed (non-IID) data impedes model convergence and performance. This paper focuses on data-dependent heterogeneity in FL and proposes a novel approach leveraging mean latent representations extracted from locally trained models. The proposed method normalizes client contributions based on these representations, allowing the central server to estimate and adjust for heterogeneity during aggregation. This normalization enhances the global model's generalization and mitigates the limitations of conventional federated averaging methods. The main contributions include introducing a normalization scheme using mean latent representations to handle statistical heterogeneity in FL, demonstrating the seamless integration with existing FL algorithms to improve performance in non-IID settings, and validating the approach through extensive experiments on diverse datasets. Results show significant improvements in model accuracy and consistency across skewed distributions. Our experiments with six FL schemes: FedAvg, FedProx, FedBABU, FedNova, SCAFFOLD, and SGDM highlight the robustness of our approach. This research advances FL by providing a practical and computationally efficient solution for statistical heterogeneity, contributing to the development of more reliable and generalized machine learning models.

What problem does this paper attempt to address?

This paper attempts to address the issue of statistical heterogeneity caused by non-independent and identically distributed (non-IID) data in Federated Learning (FL). Specifically, the paper focuses on how data dependency and statistical heterogeneity among different clients affect the convergence and performance of the model during the federated learning process. ### Background - **Federated Learning**: Federated Learning is a distributed machine learning method that allows multiple devices or clients to collaboratively train a global model without sharing data. - **Challenges**: In practical applications, the data distribution of different clients is often non-IID, which can lead to slower model convergence, reduced generalization ability, and client drift issues. ### Solution - **Proposed Method**: The paper proposes a normalization method based on mean latent representations, which normalizes the contribution values of clients by extracting the mean latent representations of local training models. - **Working Principle**: - Clients send the weights and mean latent representations of their local models to the central server. - The central server calculates the similarity between clients and adjusts each client's contribution value based on these similarities. - In this way, the central server can estimate and adjust statistical heterogeneity during the aggregation process, thereby improving the generalization ability and performance of the global model. ### Main Contributions 1. **Proposed a New Normalization Scheme**: Utilizes mean latent representations to handle statistical heterogeneity in federated learning. 2. **Seamless Integration with Existing FL Algorithms**: Demonstrates that the method can be seamlessly integrated with existing FL algorithms (such as FedAvg, FedProx, etc.) to improve performance in non-IID settings. 3. **Experimental Validation**: Extensive experiments on multiple datasets prove the effectiveness of the method, significantly improving model accuracy and consistency. ### Experimental Results - **CIFAR-10**: Experiments conducted on 50 clients show that the normalization method significantly improves performance over all baseline methods, especially in high heterogeneity settings (α=0.1). - **FEMNIST**: Experiments conducted on 10 and 20 clients show that the normalization method brings up to a 9% accuracy improvement. - **CINIC-10**: Experiments conducted on 100 clients show that the normalization method outperforms the original methods at all α values, particularly excelling in high heterogeneity settings. ### Discussion - **Universality**: The method performs well on different datasets and FL methods, especially suitable for highly heterogeneous data distributions. - **Computational Overhead**: The method introduces low computational overhead and can be seamlessly integrated into existing FL frameworks. - **Future Work**: Proposes directions for further optimization of client selection, malicious client detection, adaptive normalization techniques, and integration with privacy protection mechanisms. In summary, this paper effectively addresses the issue of statistical heterogeneity caused by non-IID data in federated learning by proposing a normalization method based on mean latent representations, providing new insights for improving the performance and reliability of federated learning systems.

Client Contribution Normalization for Enhanced Federated Learning

CDFed: Contribution-based Dynamic Federated Learning for Managing System and Statistical Heterogeneity

FedNorm: an Efficient Federated Learning Framework with Dual Heterogeneity Coexistence on Edge Intelligence Systems.

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning

Federated Learning under Heterogeneous and Correlated Client Availability

FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering

Collaboratively Learning Federated Models from Noisy Decentralized Data

Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration

Efficient Client Contribution Evaluation for Horizontal Federated Learning

FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning

Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains

Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients

No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices

Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression.

FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learning

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

Gradient Masked Averaging for Federated Learning

Client Selection for Federated Learning With Non-IID Data in Mobile Edge Computing

Enhancing generalization in federated learning with heterogeneous data: A comparative literature review