Abstract:Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data. In practice, due to the system and statistical heterogeneity among devices, synchronous FL often encounters the straggler effect. In contrast, asynchronous FL can mitigate this problem, making it suitable for scenarios involving numerous participants. However, Non-IID data and stale models present significant challenges to asynchronous FL, as they would diminish the practicality of the global model and even lead to training failures. In this work, we propose a novel asynchronous FL framework called Federated Historical Learning (FedHist), which effectively addresses the challenges posed by both Non-IID data and gradient staleness. FedHist enhances the stability of local gradients by performing weighted fusion with historical global gradients cached on the server. Relying on hindsight, it assigns aggregation weights to each participant in a multi-dimensional manner during each communication round. To further enhance the efficiency and stability of the training process, we introduce an intelligent $\ell_2$-norm amplification scheme, which dynamically regulates the learning progress based on the $\ell_2$-norms of the submitted gradients. Extensive experiments demonstrate that FedHist outperforms state-of-the-art methods in terms of convergence performance and test accuracy.

What problem does this paper attempt to address?

The paper attempts to address the challenges posed by Non-IID (Non-Independent and Identically Distributed) data and stale gradients in heterogeneous federated learning. Specifically: 1. **Non-IID Data**: In federated learning, the data distribution across different clients can vary significantly, leading to biased local gradients that affect the convergence and performance of the global model. 2. **Stale Gradients**: In asynchronous federated learning, due to network delays and uneven computational capabilities, some clients may upload outdated gradients. These stale gradients can reduce the effectiveness of the global model and even cause training to fail. To tackle these issues, the paper proposes a new asynchronous federated learning framework called Federated Historical Learning (FedHist), which enhances model stability and convergence speed through three core components: 1. **Enhancement of Gradient Stability (EGS)**: By maintaining a historical gradient buffer on the server side, suitable collaborative gradients are selected and weighted with local gradients to increase gradient diversity and reduce gradient divergence. 2. **History-Aware Aggregation (HAA)**: During the aggregation phase, not only the freshness of the gradients is considered, but additional weights are assigned based on the historical performance of the clients, thereby more comprehensively utilizing the information from different clients. 3. **Intelligent ℓ2-Norm Amplification (INA)**: By dynamically adjusting the ℓ2 norm of the aggregated gradients, the convergence process of the model is accelerated while maintaining model stability. Experimental results show that FedHist significantly outperforms existing methods in various scenarios of data heterogeneity and stale gradients, demonstrating faster convergence speed and higher test accuracy.

Take History as a Mirror in Heterogeneous Federated Learning

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance Based Adaptive Weight Aggregation

Federating from History in Streaming Federated Learning

Reschedule Gradients: Temporal Non-IID Resilient Federated Learning

Lazy Aggregation for Heterogeneous Federated Learning

FedH2L: A Federated Learning Approach with Model and Statistical Heterogeneity

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Enhancing Convergence in Federated Learning: A Contribution-Aware Asynchronous Approach

Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients

Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data

Federated mutual learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

Integrating Staleness and Shapley Value Consistency for Efficient K-Asynchronous Federated Learning

Enhancing Edge-Assisted Federated Learning with Asynchronous Aggregation and Cluster Pairing

Towards Efficient Asynchronous Federated Learning in Heterogeneous Edge Environments

FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-aware Model Update

FedLC: Accelerating Asynchronous Federated Learning in Edge Computing

Research and Implementation of Asynchronous Transmission and Update Strategy for Federated Learning

Heterogeneous Federated Learning with Cross-layer Model Fusion.

Fairness-aware loss history based federated learning heuristic algorithm

Fairness and Accuracy in Horizontal Federated Learning