Abstract:Federated Learning is designed for training models using data distributed across multiple Internet of Things(IoT) devices or servers, reducing data transfer overhead and ensuring data security. However, the decentralization and diversity of IoT devices introduce statistical and system heterogeneity, which can lead to unstable model training and even system crashes. Although many studies attribute performance issues to client-drift caused by this heterogeneity, there’s a lack of insight into how different forms of heterogeneity impact local model gradient variations and model convergence. In this paper, we investigate model gradient distribution characteristics in heterogeneous training. We find that the challenge isn’t solely due to client-drift but is also closely linked to a high degree of model overfitting, which negatively affects local model training and equilibrium convergence. To address this challenge, we introduce an efficient framework called GOFL. First, GOFL incorporates the Federated Gradient Normalization (FGN) technique to maintain gradient distribution consistency while mitigating client-drift stemming from heterogeneity. We also highlight the benefits of FGN in reducing local model overfitting and improving convergence. Secondly, GOFL introduces the Federated Device Aggregation (FDA) strategy, a critical addition to FGN. It adaptively guides device selection and aggregation based on device contributions, ensuring a more balanced training approach in the face of system heterogeneity. The experimental results demonstrate that GOFL achieves state-of-the-art training accuracy while reducing the number of training rounds. In particular, it improves the accuracy of the classical FL framework FedAvg by 30.57% and reduces the number of convergence rounds by 5.17 times.

A Hierarchical Gradient Tracking Algorithm for Mitigating Subnet-Drift in Fog Learning Networks

Taming Subnet-Drift in D2D-Enabled Fog Learning: A Hierarchical Gradient Tracking Approach

Hierarchical Federated Learning: the Interplay of User Mobility and Data Heterogeneity

Hierarchical Federated Learning with Multi-Timescale Gradient Correction

FedDGP: Disentangling Global and Personal Models for Federated Learning

DegaFL: Decentralized Gradient Aggregation for Cross-silo Federated Learning

Over-the-Air Decentralized Federated Learning

Gradient-Congruity Guided Federated Sparse Training

DRAG: Divergence-based Adaptive Aggregation in Federated Learning on Non-IID Data

Harnessing Client Drift with Decoupled Gradient Dissimilarity

Delay-Aware Hierarchical Federated Learning

Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks

Network Gradient Descent Algorithm for Decentralized Federated Learning

Decentralized Federated Learning with Gradient Tracking over Time-Varying Directed Networks

Generalized Federated Learning via Gradient Norm-Aware Minimization and Control Variables

Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees

Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach

Improving Model Consistency of Decentralized Federated Learning via Sharpness Aware Minimization and Multiple Gossip Approaches

GOFL: an Accurate and Efficient Federated Learning Framework Based on Gradient Optimization in Heterogeneous IoT Systems

Anchor Model-Based Hybrid Hierarchical Federated Learning with Overlap SGD

Asynchronous Decentralized Federated Learning for Heterogeneous Devices