Abstract:Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem--data heterogeneity in real FL systems. However, almost all existing works have to face large communication burdens and the risk of disruption if the central server fails. Only limited efforts have been used in a decentralized way but still suffers from inferior representation ability due to sharing the full model with its neighbors. Therefore, in this paper, we propose a personalized FL framework with a decentralized partial model training called DFedAlt. It personalizes the "right" components in the modern deep models by alternately updating the shared and personal parameters to train partially personalized models in a peer-to-peer manner. To further promote the shared parameters aggregation process, we propose DFedSalt integrating the local Sharpness Aware Minimization (SAM) optimizer to update the shared parameters. It adds proper perturbation in the direction of the gradient to overcome the shared model inconsistency across clients. Theoretically, we provide convergence analysis of both algorithms in the general non-convex setting for decentralized partial model training in PFL. Our experiments on several real-world data with various data partition settings demonstrate that (i) decentralized training is more suitable for partial personalization, which results in state-of-the-art (SOTA) accuracy compared with the SOTA PFL baselines; (ii) the shared parameters with proper perturbation make partial personalized FL more suitable for decentralized training, where DFedSalt achieves most competitive performance.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve several key challenges in Personalized Federated Learning (PFL): 1. **Data Heterogeneity**: In actual federated learning systems, the data distributions of different clients vary greatly, which leads to difficulties in training models. Although existing PFL methods attempt to address this issue through personalized models, they still face problems such as high communication burdens and the risk of central server failures. 2. **Communication Burden**: Traditional federated learning methods usually rely on a central server for model aggregation, which results in a large amount of communication overhead, and if the central server fails, the entire system may be interrupted. 3. **Inferior Representation Ability in Decentralized Training**: Some studies attempt to perform model aggregation in a decentralized manner, but due to sharing the entire model with neighboring nodes, the representation ability is reduced and the unique information of each client is lost. ### The methods proposed in the paper To solve the above problems, the authors propose a new framework - **DFedAlt** and its enhanced version **DFedSalt**, as follows: - **DFedAlt**: Conduct personalized training of partial models in a decentralized network by alternately updating shared parameters and personal parameters. This method allows each client to share useful feature extraction capabilities with other clients while maintaining its own uniqueness. - **DFedSalt**: Based on DFedAlt, a local Sharpness Aware Minimization (SAM) optimizer is introduced to further improve the consistency and generalization ability of shared parameters. SAM ensures the consistency of shared parameters among different clients by adding appropriate perturbations in the gradient direction. ### Main contributions 1. **Proposing a new decentralized partial - model personalization framework**: By decomposing the local model into a shared part and a personal part and alternately optimizing them in a decentralized environment, the representation ability and performance of the model are improved. 2. **Enhancing the consistency of shared parameters**: By integrating the SAM optimizer, the model inconsistency caused by data heterogeneity is reduced, and the generalization ability of the overall model is improved. 3. **Providing theoretical analysis**: For the first time, convergence analysis of decentralized partial - model personalization is carried out, proving the effectiveness of this method in non - convex settings. 4. **Extensive experimental verification**: A large number of experiments are carried out on multiple real - world datasets to verify the superior performance of the proposed method under different data partitioning methods, especially outstanding in extreme data heterogeneity scenarios. Through these contributions, the paper demonstrates the potential of decentralized partial - model personalization in federated learning and provides new ideas and methods for future research.

Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Decentralized Directed Collaboration for Personalized Federated Learning

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training

Achieving Personalized Federated Learning with Sparse Local Models

Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity

PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning

Personalizing or Not: Dynamically Personalized Federated Learning with Incentives

Group Personalized Federated Learning

FedReMa: Improving Personalized Federated Learning via Leveraging the Most Relevant Clients

Personalized Edge Intelligence via Federated Self-Knowledge Distillation

Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models

Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer

Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity

Sparse Personalized Federated Learning

Personalized Federated Learning with First Order Model Optimization

Sparse Federated Learning with Hierarchical Personalized Models

Dual Calibration-based Personalised Federated Learning

Personalized Federated Learning With Differential Privacy and Convergence Guarantee

Efficient Personalized Federated Learning via Sparse Model-Adaptation

Client-supervised Federated Learning: Towards One-model-for-all Personalization