Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training

Yifan Shi,Yingqi Liu,Yan Sun,Zihao Lin,Li Shen,Xueqian Wang,Dacheng Tao
2023-05-24
Abstract:Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem--data heterogeneity in real FL systems. However, almost all existing works have to face large communication burdens and the risk of disruption if the central server fails. Only limited efforts have been used in a decentralized way but still suffers from inferior representation ability due to sharing the full model with its neighbors. Therefore, in this paper, we propose a personalized FL framework with a decentralized partial model training called DFedAlt. It personalizes the "right" components in the modern deep models by alternately updating the shared and personal parameters to train partially personalized models in a peer-to-peer manner. To further promote the shared parameters aggregation process, we propose DFedSalt integrating the local Sharpness Aware Minimization (SAM) optimizer to update the shared parameters. It adds proper perturbation in the direction of the gradient to overcome the shared model inconsistency across clients. Theoretically, we provide convergence analysis of both algorithms in the general non-convex setting for decentralized partial model training in PFL. Our experiments on several real-world data with various data partition settings demonstrate that (i) decentralized training is more suitable for partial personalization, which results in state-of-the-art (SOTA) accuracy compared with the SOTA PFL baselines; (ii) the shared parameters with proper perturbation make partial personalized FL more suitable for decentralized training, where DFedSalt achieves most competitive performance.
Machine Learning,Distributed, Parallel, and Cluster Computing,Optimization and Control
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to solve several key challenges in Personalized Federated Learning (PFL): 1. **Data Heterogeneity**: In actual federated learning systems, the data distributions of different clients vary greatly, which leads to difficulties in training models. Although existing PFL methods attempt to address this issue through personalized models, they still face problems such as high communication burdens and the risk of central server failures. 2. **Communication Burden**: Traditional federated learning methods usually rely on a central server for model aggregation, which results in a large amount of communication overhead, and if the central server fails, the entire system may be interrupted. 3. **Inferior Representation Ability in Decentralized Training**: Some studies attempt to perform model aggregation in a decentralized manner, but due to sharing the entire model with neighboring nodes, the representation ability is reduced and the unique information of each client is lost. ### The methods proposed in the paper To solve the above problems, the authors propose a new framework - **DFedAlt** and its enhanced version **DFedSalt**, as follows: - **DFedAlt**: Conduct personalized training of partial models in a decentralized network by alternately updating shared parameters and personal parameters. This method allows each client to share useful feature extraction capabilities with other clients while maintaining its own uniqueness. - **DFedSalt**: Based on DFedAlt, a local Sharpness Aware Minimization (SAM) optimizer is introduced to further improve the consistency and generalization ability of shared parameters. SAM ensures the consistency of shared parameters among different clients by adding appropriate perturbations in the gradient direction. ### Main contributions 1. **Proposing a new decentralized partial - model personalization framework**: By decomposing the local model into a shared part and a personal part and alternately optimizing them in a decentralized environment, the representation ability and performance of the model are improved. 2. **Enhancing the consistency of shared parameters**: By integrating the SAM optimizer, the model inconsistency caused by data heterogeneity is reduced, and the generalization ability of the overall model is improved. 3. **Providing theoretical analysis**: For the first time, convergence analysis of decentralized partial - model personalization is carried out, proving the effectiveness of this method in non - convex settings. 4. **Extensive experimental verification**: A large number of experiments are carried out on multiple real - world datasets to verify the superior performance of the proposed method under different data partitioning methods, especially outstanding in extreme data heterogeneity scenarios. Through these contributions, the paper demonstrates the potential of decentralized partial - model personalization in federated learning and provides new ideas and methods for future research.