Abstract:Personalized Federated Learning (PFL) relies on collective data knowledge to build customized models. However, non-IID data between clients poses significant challenges, as collaborating with clients who have diverse data distributions can harm local model performance, especially with limited training data. To address this issue, we propose FedACS, a new PFL algorithm with an Attention-based Client Selection mechanism. FedACS integrates an attention mechanism to enhance collaboration among clients with similar data distributions and mitigate the data scarcity issue. It prioritizes and allocates resources based on data similarity. We further establish the theoretical convergence behavior of FedACS. Experiments on CIFAR10 and FMNIST validate FedACS's superiority, showcasing its potential to advance personalized federated learning. By tackling non-IID data challenges and data scarcity, FedACS offers promising advances in the field of personalized federated learning.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the performance degradation of local models in personalized federated learning (PFL) due to the non - independent and identically distributed (non - IID) characteristics of client - side data distributions. Specifically, when clients participating in training have different data distributions, this diversity may harm the performance of local models, especially when the training data is limited. To address this challenge, the authors propose the FEDACS algorithm, a new PFL algorithm with an attention - based client - selection method. FEDACS alleviates the data scarcity problem by enhancing the collaboration between clients with similar data distributions and preferentially allocating resources according to data similarity, thereby improving the effectiveness of personalized federated learning.
The key points of the paper include:
- **Problem Definition**: In PFL, the goal is to train personalized models \(w_1, w_2, \ldots, w_n\) while respecting data privacy and meeting specific user needs. The optimization problem can be expressed as:
\[
w^*_1, w^*_2, \ldots, w^*_n=\arg \min_{w_1, \ldots, w_n} \sum_{i = 1}^n F_i(w_i)
\]
where \(F_i(w_i)\) represents the expected loss on the data distribution of the \(i\)-th client.
- **Proposed Method**: The FEDACS algorithm promotes cooperation between clients with similar data distributions by introducing a regularization term \(R(W)\). The optimization problem can be expressed as:
\[
\min_W F_\lambda(W)=F(W)+\lambda R(W)=F(W)+\lambda \sum_{i,j = 1}^n s_{ij}\|w_i - w_j\|^2
\]
where \(W = [w_1, \ldots, w_n]\) is a matrix containing client - side local models, \(\lambda\) is a regularization parameter, and \(s_{ij}\) is a normalized score used to measure the similarity of data distributions.
- **Algorithm Flow**:
1. Initialize client - side models \(w^0_1, w^0_2, \ldots, w^0_n\).
2. For each round \(k\):
- The server randomly selects clients to participate.
- Calculate the model similarity matrix \(S\).
- Calculate the \(p\)-quantile of model similarity as the threshold \(\delta\).
- The server updates the intermediate models \(u^k_1, u^k_2, \ldots, u^k_n\).
- Each client \(i\) updates its local model \(w^k_i\).
- **Experimental Results**: Experiments were conducted on the CIFAR10 and FMNIST datasets, and the results show that FEDACS outperforms other methods in handling non - IID data and data scarcity problems, especially when the data is insufficient.
In conclusion, FEDACS effectively solves the data heterogeneity and data scarcity problems in personalized federated learning by introducing an attention - based client - selection method, improving the generalization ability and performance of the model.