Abstract:In practical federated learning (FL) systems, the presence of malicious Byzantine attacks and data heterogeneity often introduces biases into the learning process. However, existing Byzantine-robust methods typically only achieve a compromise between adaptability to different loss function types (including both strongly convex and non-convex) and robustness to heterogeneous datasets, but with non-zero optimality gap. Moreover, this compromise often comes at the cost of high computational complexity for aggregation, which significantly slows down the training speed. To address this challenge, we propose a federated learning approach called Federated Normalized Gradients Algorithm (Fed-NGA). Fed-NGA simply normalizes the uploaded local gradients to be unit vectors before aggregation, achieving a time complexity of $\mathcal{O}(pM)$, where $p$ represents the dimension of model parameters and $M$ is the number of participating clients. This complexity scale achieves the best level among all the existing Byzantine-robust methods. Furthermore, through rigorous proof, we demonstrate that Fed-NGA transcends the trade-off between adaptability to loss function type and data heterogeneity and the limitation of non-zero optimality gap in existing literature. Specifically, Fed-NGA can adapt to both non-convex loss functions and non-IID datasets simultaneously, with zero optimality gap at a rate of $\mathcal{O} (1/T^{\frac{1}{2} - \delta})$, where T is the iteration number and $\delta \in (0,\frac{1}{2})$. In cases where the loss function is strongly convex, the zero optimality gap achieving rate can be improved to be linear. Experimental results provide evidence of the superiority of our proposed Fed-NGA on time complexity and convergence performance over baseline methods.

Generalized Federated Learning via Gradient Norm-Aware Minimization and Control Variables

FedDGP: Disentangling Global and Personal Models for Federated Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

FedGAMMA: Federated Learning with Global Sharpness-Aware Minimization.

Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression.

Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learning

Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating

Byzantine-resilient Federated Learning Employing Normalized Gradients on Non-IID Datasets

Sparse Personalized Federated Learning

Lazy Aggregation for Heterogeneous Federated Learning

Gradient Rotation Unit for Non-I.I.D. Federated Learning

Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape

Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning

Pfedgf: Enabling Personalized Federated Learning Via Gradient Fusion

Enhancing Generalization Robustness of Federated Learning in Highly Heterogeneous Environments

GRP-FED: Addressing Client Imbalance in Federated Learning via Global-Regularized Personalization

Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization

FedAgg: Adaptive Federated Learning with Aggregated Gradients

Rethinking Client Drift in Federated Learning: A Logit Perspective

Adaptive Federated Learning on Non-IID Data with Resource Constraint

FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure