Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity

Zheng Wang,Zheng Wang,Zhaopeng Peng,Zihui Wang,Cheng Wang

2024-05-31

Abstract:Federated Learning (FL) stands to gain significant advantages from collaboratively training capacity-heterogeneous models, enabling the utilization of private data and computing power from low-capacity devices. However, the focus on personalizing capacity-heterogeneous models based on client-specific data has been limited, resulting in suboptimal local model utility, particularly for low-capacity clients. The heterogeneity in both data and device capacity poses two key challenges for model personalization: 1) accurately retaining necessary knowledge embedded within reduced submodels for each client, and 2) effectively sharing knowledge through aggregating size-varying parameters. To this end, we introduce Pa3dFL, a novel framework designed to enhance local model performance by decoupling and selectively sharing knowledge among capacity-heterogeneous models. First, we decompose each layer of the model into general and personal parameters. Then, we maintain uniform sizes for the general parameters across clients and aggregate them through direct averaging. Subsequently, we employ a hyper-network to generate size-varying personal parameters for clients using learnable embeddings. Finally, we facilitate the implicit aggregation of personal parameters by aggregating client embeddings through a self-attention module. We conducted extensive experiments on three datasets to evaluate the effectiveness of Pa3dFL. Our findings indicate that Pa3dFL consistently outperforms baseline methods across various heterogeneity settings. Moreover, Pa3dFL demonstrates competitive communication and computation efficiency compared to baseline approaches, highlighting its practicality and adaptability in adverse system conditions.

Machine Learning,Artificial Intelligence,Distributed, Parallel, and Cluster Computing

What problem does this paper attempt to address?

This paper attempts to solve the problem of how to train personalized federated learning models under the conditions of device - capacity heterogeneity and personal data differences. Specifically, it focuses on two key challenges:

1. **Accurately Retain Necessary Knowledge**: When pruning the model for each client, how to accurately retain the necessary knowledge embedded in the reduced sub - models.
2. **Effectively Aggregate Knowledge**: How to effectively aggregate knowledge across clients between models of different capacities.

To address these challenges, the authors propose a new framework named Pa3dFL. This framework enhances the performance of local models through the following steps:

1. **Layer Decomposition**: Decompose each layer of the model's parameters into general parameters and personalized parameters. The general parameters remain the same size across all clients and are aggregated by direct averaging; while the personalized parameters are generated according to the specific needs of the clients.
2. **Hyper - network Generates Personalized Parameters**: Use a hyper - network to generate personalized parameters of different sizes from learnable client embeddings.
3. **Implicit Aggregation by Self - attention Module**: Aggregate client embeddings through the self - attention module, thereby promoting personalized knowledge sharing among similar clients.

Experimental results show that Pa3dFL outperforms baseline methods in a variety of heterogeneous settings, not only performing well in terms of accuracy, but also being competitive in terms of communication and computational efficiency. This indicates that Pa3dFL has high practicality and adaptability in practical applications.