Abstract:Federated learning encounters a critical challenge of data heterogeneity, adversely affecting the performance and convergence of the federated model. Various approaches have been proposed to address this issue, yet their effectiveness is still limited. Recent studies have revealed that the federated model suffers severe forgetting in local training, leading to global forgetting and performance degradation. Although the analysis provides valuable insights, a comprehensive understanding of the vulnerable classes and their impact factors is yet to be established. In this paper, we aim to bridge this gap by systematically analyzing the forgetting degree of each class during local training across different communication rounds. Our observations are: (1) Both missing and non-dominant classes suffer similar severe forgetting during local training, while dominant classes show improvement in performance. (2) When dynamically reducing the sample size of a dominant class, catastrophic forgetting occurs abruptly when the proportion of its samples is below a certain threshold, indicating that the local model struggles to leverage a few samples of a specific class effectively to prevent forgetting. Motivated by these findings, we propose a novel and straightforward algorithm called Federated Knowledge Anchor (FedKA). Assuming that all clients have a single shared sample for each class, the knowledge anchor is constructed before each local training stage by extracting shared samples for missing classes and randomly selecting one sample per class for non-dominant classes. The knowledge anchor is then utilized to correct the gradient of each mini-batch towards the direction of preserving the knowledge of the missing and non-dominant classes. Extensive experimental results demonstrate that our proposed FedKA achieves fast and stable convergence, significantly improving accuracy on popular benchmarks.

Flashback: Understanding and Mitigating Forgetting in Federated Learning

Accurate Forgetting for Heterogeneous Federated Continual Learning

FedME2: Memory Evaluation & Erase Promoting Federated Unlearning in DTMN

Knowledge Lock: Overcoming Catastrophic Forgetting in Federated Learning

Acceleration of Federated Learning with Alleviated Forgetting in Local Training

Overcoming Forgetting in Federated Learning on Non-IID Data

FLASH: Federated Learning Across Simultaneous Heterogeneities

Continual Federated Learning Based on Knowledge Distillation

Decentralized Federated Learning under Communication Delays.

GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting

No One Left Behind: Real-World Federated Class-Incremental Learning

TARGET: Federated Class-Continual Learning Via Exemplar-Free Distillation

On Knowledge Editing in Federated Learning: Perspectives, Challenges, and Future Directions

Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor

Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory

Dmaf: Data-Model Anti-Forgetting for Federated Incremental Learning

Towards Efficient Replay in Federated Incremental Learning

Cyclical Weight Consolidation: Towards Solving Catastrophic Forgetting in Serial Federated Learning

HiFlash: Communication-Efficient Hierarchical Federated Learning with Adaptive Staleness Control and Heterogeneity-aware Client-Edge Association

Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning

KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting