FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfer

Xin Gao,Xin Yang,Hao Yu,Yan Kang,Tianrui Li
2024-05-04
Abstract:Federated Class-Incremental Learning (FCIL) focuses on continually transferring the previous knowledge to learn new classes in dynamic Federated Learning (FL). However, existing methods do not consider the trustworthiness of FCIL, i.e., improving continual utility, privacy, and efficiency simultaneously, which is greatly influenced by catastrophic forgetting and data heterogeneity among clients. To address this issue, we propose FedProK (Federated Prototypical Feature Knowledge Transfer), leveraging prototypical feature as a novel representation of knowledge to perform spatial-temporal knowledge transfer. Specifically, FedProK consists of two components: (1) feature translation procedure on the client side by temporal knowledge transfer from the learned classes and (2) prototypical knowledge fusion on the server side by spatial knowledge transfer among clients. Extensive experiments conducted in both synchronous and asynchronous settings demonstrate that our FedProK outperforms the other state-of-the-art methods in three perspectives of trustworthiness, validating its effectiveness in selectively transferring spatial-temporal knowledge.
Machine Learning,Artificial Intelligence,Neural and Evolutionary Computing
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to achieve trustworthy Federated Class - Incremental Learning (FCIL) in the Federated Learning (FL) environment. Specifically, the paper focuses on how to simultaneously improve continuous utility, privacy protection, and efficiency in dynamic and heterogeneous Federated Learning scenarios, especially when facing two key challenges: catastrophic forgetting and data heterogeneity. ### Background and Problem Description **Background**: - **Federated Learning (FL)**: A distributed machine - learning method that allows multiple clients to collaborate in training a model without sharing data. - **Class - Incremental Learning (CIL)**: A continuous - learning method aimed at enabling the model to learn new knowledge without forgetting old knowledge when new classes appear. **Problem**: - **Catastrophic Forgetting**: When the model is learning a new task, it is prone to forget the knowledge of previously learned tasks. - **Data Heterogeneity**: The data distributions of different clients vary greatly, resulting in a decline in the performance of the global model and unstable convergence. ### Limitations of Existing Methods - **Storing or Generating Samples**: Existing methods usually relieve catastrophic forgetting by storing or generating samples of old classes and replaying these samples when training new tasks. However, this method is limited by the size of the sample set and may exceed the memory capacity of edge devices. - **Communication Costs and Privacy Issues**: Some methods generate pseudo - samples on the server and distribute them to clients, which will increase communication costs and introduce potential security risks. - **Balancing Stability and Plasticity**: Existing FCIL methods fail to well balance the model's retention of old knowledge (stability) and its adaptation to new knowledge (plasticity). ### The Solution Proposed in the Paper To address the above problems, the paper proposes **FedProK (Federated Prototypical Feature Knowledge Transfer)**, a trustworthy Federated Class - Incremental Learning framework. FedProK achieves spatio - temporal knowledge transfer through the following two components: 1. **Feature Translation**: - **Temporal Knowledge Transfer**: On the client side, the feature translation mechanism is used to transfer the features of old classes to new classes to alleviate catastrophic forgetting. - **Feature Translation Process**: \[ \hat{f}_t(x_p)=f(x_n)+\mu_t^{k,p}-\mu_t^{k,n} \] where \( f(x_n) \) is the real - sample feature of the new class \( n \), and \( \mu_t^{k,p} \) and \( \mu_t^{k,n} \) are the prototypes of the old class \( p \) and the new class \( n \), respectively. 2. **Prototypical Knowledge Fusion**: - **Spatial Knowledge Transfer**: On the server side, the prototypical knowledge fusion mechanism is used to transfer knowledge among clients to solve the problem of data heterogeneity. - **Fusion Process**: \[ \mu_t^c = \begin{cases} \frac{\sum_{k = 1}^K|D_k^c|\mu_r^{k,c}}{|D_c|}, & \text{if } c\notin\cup_{i = 1}^{t - 1}C_i \\ \beta\frac{\sum_{k = 1}^K|D_k^c|\mu_r^{k,c}}{|D_c|}+(1 - \beta)\mu_{t - 1}^c \end{cases} \]