Abstract:As federated learning (FL) continues to advance, research has branched into two major directions: enhancing the individual global model and developing multiple personalized models. However, few studies took both orientations into consideration in parallel. The underlying reason lies in the objective heterogeneity between global and personalized models, where the former emphasizes generalization, while the latter focuses on specialization. The dilemma of sacrificing the personalized deviation in exchange for stable global enhancement, or forsaking the global model for personalization, is particularly acute in highly heterogeneous data scenarios (as a curse). To establish the coexistence of global and personalized models in highly heterogeneous data scenarios, we propose FedAKD, a generalized federated learning framework with adaptive knowledge distillation to optimizes both global and personalized models. Specifically, based on the similarity of predictive distributions over a reference dataset, two distinct adaptive weighting strategies are employed to match the divergent focuses of global and personalized optimization, leveraging data heterogeneity to foster cooperative adaptation and establish their coexistence (as a blessing). The client-side strategy through cluster-oriented optimization to facilitate the domain adaptation in local tasks. Meanwhile, the server-side strategy adjusts the global model to accommodate varying optimization directions from diverse implicit clusters. Experiments in label and feature shift settings demonstrate that our method outperforms state-of-the-art methods in both global and personalized performance with faster convergence [e.g. the accuracy improvements on the physical activity monitoring dataset (PAMAP2) for global and personalized models exceed 26.51% and 11.00%, respectively]. Furthermore, our method is equally applicable to real-world healthcare scenarios.

Learning to Generalize in Heterogeneous Federated Networks

FedDGP: Disentangling Global and Personal Models for Federated Learning

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning

Improving Global Generalization and Local Personalization for Federated Learning

An Efficient and Security Federated Learning for Data Heterogeneity

FedGKD: Towards Heterogeneous Federated Learning via Global Knowledge Distillation

FEDGKD: Toward Heterogeneous Federated Learning Via Global Knowledge Distillation

Towards Efficient Model-Heterogeneity Federated Learning for Large Models

Federated Graph Learning under Domain Shift with Generalizable Prototypes

Personalized Federated Learning with Layer-Wise Feature Transformation Via Meta-Learning

Exploiting Feature Heterogeneity for Improved Generalization in Federated Multi-task Learning

To Be Global or Personalized: Generalized Federated Learning with Cooperative Adaptation for Data Heterogeneity

A Unified Personalized Federated Learning Framework Ensuring Domain Generalization

FedPer++: Toward Improved Personalized Federated Learning on Heterogeneous and Imbalanced Data

Enhancing Generalization Robustness of Federated Learning in Highly Heterogeneous Environments

Tackling Data Heterogeneity in Federated Learning with Class Prototypes

Federated Domain Generalization with Label Smoothing and Balanced Decentralized Training

Heterogeneous Federated Meta-Learning with Mutually Constrained Propagation

FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

Privacy preserving federated learning for full heterogeneity

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning