Abstract:To mitigate the privacy leakages and reduce the communication burden of Federated Learning (FL), decentralized FL (DFL) discards the central server and each client only communicates with its neighbors in the decentralized communication network. However, existing DFL algorithms tend to feature high inconsistency among local models, which results in severe distribution shifts across clients and inferior performance compared with centralized FL (CFL), especially on heterogeneous data or with sparse connectivity of communication topology. To alleviate this challenge, we propose two DFL algorithms named DFedSAM and DFedSAM-MGS to improve the performance. Specifically, DFedSAM leverages gradient perturbation to generate local flatness models via Sharpness Aware Minimization (SAM), which searches for model parameters with uniformly low loss function values. In addition, DFedSAM-MGS further boosts DFedSAM by adopting the technique of Multiple Gossip Steps (MGS) for a better model consistency, which accelerates the aggregation of local flatness models and better balances the communication complexity and learning performance. In the theoretical perspective, we present the improved convergence rates $\small \mathcal{O}\big(\frac{1}{T}+\frac{1}{T^2(1-\lambda)^2}\big)$ and $\small \mathcal{O}\big(\frac{1}{T}+\frac{\lambda^Q+1}{T^2(1-\lambda^Q)^2}\big)$ in the stochastic non-convex setting for DFedSAM and DFedSAM-MGS, respectively, where $1-\lambda$ is the spectral gap of the gossip matrix $W$ and $Q$ is the gossip steps in MGS. Meanwhile, we empirically confirm that our methods can achieve competitive performance compared with CFL baselines and outperform existing DFL baselines.

FedGAMMA: Federated Learning with Global Sharpness-Aware Minimization.

FedDGP: Disentangling Global and Personal Models for Federated Learning

Generalized Federated Learning via Gradient Norm-Aware Minimization and Control Variables

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape

Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning

Improving Model Consistency of Decentralized Federated Learning via Sharpness Aware Minimization and Multiple Gossip Approaches

Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awareness

Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization

Rethinking Client Drift in Federated Learning: A Logit Perspective

FedGH: Heterogeneous Federated Learning with Generalized Global Header

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

FedAgg: Adaptive Federated Learning with Aggregated Gradients

FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning

Improving the Model Consistency of Decentralized Federated Learning

FedGK: Communication-Efficient Federated Learning through Group-Guided Knowledge Distillation

FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients

Acceleration of Federated Learning with Alleviated Forgetting in Local Training

PGFed: Personalize Each Client's Global Objective for Federated Learning

Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy

FedImpro: Measuring and Improving Client Update in Federated Learning