Abstract:In order to address the problem of data heterogeneity, in recent years, personalized federated learning has tailored models to individual user data to enhance model performance on clients with diverse data distributions. However, the existing personalized federated learning methods do not adequately address the problem of data heterogeneity, and lack the processing of system heterogeneity. Consequently, these issues lead to diminished training efficiency and suboptimal model performance of personalized federated learning in heterogeneous environments. In response to these challenges, we propose FedPRL, a novel approach to personalized federated learning designed specifically for heterogeneous environments. Our method tackles data heterogeneity by implementing a personalized strategy centered on local data storage, enabling the accurate extraction of features tailored to the data distribution of individual clients. This personalized approach enhances the performance of federated learning models when dealing with non-IID data. To overcome system heterogeneity, we design a client selection mechanism grounded in reinforcement learning and user quality evaluation. This mechanism optimizes the selection of clients based on data quality and training time, thereby boosting the efficiency of the training process and elevating the overall performance of personalized models. Moreover, we devise a local training method that utilizes global knowledge distillation of non-target classes, which combined with traditional federated learning can effectively address the issue of catastrophic forgetting during global model updates. This approach enhances the generalization capability of the global model and further improves the performance of personalized models. Extensive experiments on both standard and real-world datasets demonstrate that FedPRL effectively resolves the challenges of data and system heterogeneity, enhancing the efficiency and model performance of personalized federated learning methods in heterogeneous environments, and outperforming state-of-the-art methods in terms of model accuracy and training efficiency.

Federated Reinforcement Learning with Constraint Heterogeneity

Federated Reinforcement Learning with Environment Heterogeneity

Convergence Rate of Primal-Dual Approach to Constrained Reinforcement Learning with Softmax Policy

Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments

Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis

FedHQL: Federated Heterogeneous Q-Learning

On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning

Personalized Federated Reinforcement Learning: Balancing Personalization and Experience Sharing Via Distance Constraint

Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneity

Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

Federated Offline Policy Optimization with Dual Regularization

An Efficient Personalized Federated Learning Approach in Heterogeneous Environments: a Reinforcement Learning Perspective

The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication.

Client Selection for Federated Policy Optimization with Environment Heterogeneity

FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices

Embracing Multiheterogeneity and Privacy Security Simultaneously: A Dynamic Privacy-Aware Federated Reinforcement Learning Approach

On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments