KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting

Zeke Xia,Ming Hu,Dengke Yan,Ruixuan Liu,Anran Li,Xiaofei Xie,Mingsong Chen

2024-04-19

Abstract:Although Split Federated Learning (SFL) is good at enabling knowledge sharing among resource-constrained clients, it suffers from the problem of low training accuracy due to the neglect of data heterogeneity and catastrophic forgetting. To address this issue, we propose a novel SFL approach named KoReA-SFL, which adopts a multi-model aggregation mechanism to alleviate gradient divergence caused by heterogeneous data and a knowledge replay strategy to deal with catastrophic forgetting. Specifically, in KoReA-SFL cloud servers (i.e., fed server and main server) maintain multiple branch model portions rather than a global portion for local training and an aggregated master-model portion for knowledge sharing among branch portions. To avoid catastrophic forgetting, the main server of KoReA-SFL selects multiple assistant devices for knowledge replay according to the training data distribution of each server-side branch-model portion. Experimental results obtained from non-IID and IID scenarios demonstrate that KoReA-SFL significantly outperforms conventional SFL methods (by up to 23.25\% test accuracy improvement).

Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the issue of low training accuracy in Split Federated Learning (SFL) caused by data heterogeneity and catastrophic forgetting. Specifically: 1. **Data Heterogeneity Issue**: When sharing knowledge among resource-constrained clients, the inconsistency in data distribution (non-IID) across clients can lead to gradient divergence, thereby affecting the performance of the global model. 2. **Catastrophic Forgetting Issue**: In SFL, the server only selects a subset of clients to participate in model training, which can cause the model to forget previously learned knowledge. This issue is particularly severe in non-IID data scenarios. To address these issues, the paper proposes a new SFL method called KoReA-SFL, which mitigates gradient divergence caused by data heterogeneity through a multi-model aggregation mechanism and addresses catastrophic forgetting using a knowledge replay strategy. The specific implementations include: - **Multi-Model Aggregation Mechanism**: The cloud server (i.e., the federated server and the main server) maintains multiple branch model parts instead of a single global model part for local training and knowledge sharing. - **Knowledge Replay Strategy**: The main server selects multiple auxiliary devices for knowledge replay based on the training data distribution of each branch model part to avoid catastrophic forgetting. Experimental results show that KoReA-SFL significantly outperforms traditional SFL methods in both non-IID and IID scenarios, with test accuracy improvements of up to 23.25%.

KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting

Towards Efficient Replay in Federated Incremental Learning

SR-FDIL: Synergistic Replay for Federated Domain-Incremental Learning

Have Your Cake and Eat It Too: Toward Efficient and Accurate Split Federated Learning

HSFL: an Efficient Split Federated Learning Framework Via Hierarchical Organization.

Continual Federated Learning Based on Knowledge Distillation

KDRSFL: A knowledge distillation resistance transfer framework for defending model inversion attacks in split federated learning

Knowledge Lock: Overcoming Catastrophic Forgetting in Federated Learning

Accurate Forgetting for Heterogeneous Federated Continual Learning

Splitfed learning without client-side synchronization: Analyzing client-side split network portion size to overall performance

Federated Split Learning Via Mutual Knowledge Distillation

Acceleration of Federated Learning with Alleviated Forgetting in Local Training

No One Left Behind: Real-World Federated Class-Incremental Learning

Robust Model Aggregation for Heterogeneous Federated Learning: Analysis and Optimizations

Addressing Catastrophic Forgetting in Federated Learning on Resource-Constrained Devices: A Feature Replay Approach

SplitFed resilience to packet loss: Where to split, that is the question

Robust Heterogeneous Federated Learning Via Data-Free Knowledge Amalgamation.

AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networks

RRA-FFSCIL: Inter-intra classes representation and relationship augmentation federated few-shot incremental learning

Adaptive Federated Learning on Non-IID Data with Resource Constraint

Split Aggregation: Lightweight Privacy-Preserving Federated Learning Resistant to Byzantine Attacks