Abstract:To improve the efficiency of reinforcement learning (RL), we propose a novel asynchronous federated reinforcement learning (FedRL) framework termed AFedPG, which constructs a global model through collaboration among $N$ agents using policy gradient (PG) updates. To address the challenge of lagged policies in asynchronous settings, we design a delay-adaptive lookahead technique \textit{specifically for FedRL} that can effectively handle heterogeneous arrival times of policy gradients. We analyze the theoretical global convergence bound of AFedPG, and characterize the advantage of the proposed algorithm in terms of both the sample complexity and time complexity. Specifically, our AFedPG method achieves $O(\frac{{\epsilon}^{-2.5}}{N})$ sample complexity for global convergence at each agent on average. Compared to the single agent setting with $O(\epsilon^{-2.5})$ sample complexity, it enjoys a linear speedup with respect to the number of agents. Moreover, compared to synchronous FedPG, AFedPG improves the time complexity from $O(\frac{t_{\max}}{N})$ to $O({\sum_{i=1}^{N} \frac{1}{t_{i}}})^{-1}$, where $t_{i}$ denotes the time consumption in each iteration at agent $i$, and $t_{\max}$ is the largest one. The latter complexity $O({\sum_{i=1}^{N} \frac{1}{t_{i}}})^{-1}$ is always smaller than the former one, and this improvement becomes significant in large-scale federated settings with heterogeneous computing powers ($t_{\max}\gg t_{\min}$). Finally, we empirically verify the improved performance of AFedPG in four widely-used MuJoCo environments with varying numbers of agents. We also demonstrate the advantages of AFedPG in various computing heterogeneity scenarios.

A Novel Federated Reinforcement Learning Algorithm with Historical Model Update Momentum

Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency

Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments

Towards Cost-Efficient Federated Multi-agent RL with Learnable Aggregation

Federated Model Search Via Reinforcement Learning

Reinforcement Learning As a Catalyst for Robust and Fair Federated Learning: Deciphering the Dynamics of Client Contributions

Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning

Efficient Asynchronous Federated Learning with Prospective Momentum Aggregation and Fine-Grained Correction

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning

Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis

Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning

FedMC: Federated Reinforcement Learning on the Edge with Meta-Critic Networks

The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication.

Experience-Driven Computational Resource Allocation of Federated Learning by Deep Reinforcement Learning

A Fair Federated Learning Framework with Reinforcement Learning.

Rethinking the initialization of Momentum in Federated Learning with Heterogeneous Data

Single-Loop Federated Actor-Critic across Heterogeneous Environments

Faster On-Device Training Using New Federated Momentum Algorithm

Momentum Benefits Non-iid Federated Learning Simply and Provably

On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations

A Selective Federated Reinforcement Learning Strategy for Autonomous Driving.