Abstract:Federated learning encounters a critical challenge of data heterogeneity, adversely affecting the performance and convergence of the federated model. Various approaches have been proposed to address this issue, yet their effectiveness is still limited. Recent studies have revealed that the federated model suffers severe forgetting in local training, leading to global forgetting and performance degradation. Although the analysis provides valuable insights, a comprehensive understanding of the vulnerable classes and their impact factors is yet to be established. In this paper, we aim to bridge this gap by systematically analyzing the forgetting degree of each class during local training across different communication rounds. Our observations are: (1) Both missing and non-dominant classes suffer similar severe forgetting during local training, while dominant classes show improvement in performance. (2) When dynamically reducing the sample size of a dominant class, catastrophic forgetting occurs abruptly when the proportion of its samples is below a certain threshold, indicating that the local model struggles to leverage a few samples of a specific class effectively to prevent forgetting. Motivated by these findings, we propose a novel and straightforward algorithm called Federated Knowledge Anchor (FedKA). Assuming that all clients have a single shared sample for each class, the knowledge anchor is constructed before each local training stage by extracting shared samples for missing classes and randomly selecting one sample per class for non-dominant classes. The knowledge anchor is then utilized to correct the gradient of each mini-batch towards the direction of preserving the knowledge of the missing and non-dominant classes. Extensive experimental results demonstrate that our proposed FedKA achieves fast and stable convergence, significantly improving accuracy on popular benchmarks.

Knowledge Lock: Overcoming Catastrophic Forgetting in Federated Learning

Continual Federated Learning Based on Knowledge Distillation

Acceleration of Federated Learning with Alleviated Forgetting in Local Training

Accurate Forgetting for Heterogeneous Federated Continual Learning

Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor

TARGET: Federated Class-Continual Learning Via Exemplar-Free Distillation

Overcoming Spatial-Temporal Catastrophic Forgetting for Federated Class-Incremental Learning

Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning

Federated Class-Incremental Learning with New-Class Augmented Self-Distillation

Flashback: Understanding and Mitigating Forgetting in Federated Learning

FedSKF: Selective Knowledge Fusion via Optimal Transport in Federated Class Incremental Learning

Personalized and privacy-enhanced federated learning framework via knowledge distillation

Rethinking Personalized Federated Learning from Knowledge Perspective

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

On Knowledge Editing in Federated Learning: Perspectives, Challenges, and Future Directions

Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

pFedKT: Personalized federated learning with dual knowledge transfer

Overcoming Forgetting in Federated Learning on Non-IID Data

Decentralized Federated Learning Via Mutual Knowledge Distillation.

FedINC: an Exemplar-Free Continual Federated Learning Framework with Small Labeled Data