Federated Offline Policy Optimization with Dual Regularization

Sheng Yue,Zerui Qin,Xingyuan Hua,Yongheng Deng,Ju Ren

2024-05-29

Abstract:Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interactions with the environment during local updating, which can be prohibitively expensive or even infeasible in many real-world domains. To overcome this challenge, this paper proposes a novel offline federated policy optimization algorithm, named $\texttt{DRPO}$, which enables distributed agents to collaboratively learn a decision policy only from private and static data without further environmental interactions. $\texttt{DRPO}$ leverages dual regularization, incorporating both the local behavioral policy and the global aggregated policy, to judiciously cope with the intrinsic two-tier distributional shifts in offline FRL. Theoretical analysis characterizes the impact of the dual regularization on performance, demonstrating that by achieving the right balance thereof, $\texttt{DRPO}$ can effectively counteract distributional shifts and ensure strict policy improvement in each federative learning round. Extensive experiments validate the significant performance gains of $\texttt{DRPO}$ over baseline methods.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address a key issue in Federated Reinforcement Learning (FRL): during local updates, agents need to repeatedly interact with the environment to collect online experience data, which is both expensive and impractical in many real-world applications. Specifically, existing FRL methods often require agents to interact with the environment repeatedly during local updates, a practice that can be highly inefficient or even unfeasible in practical applications, especially in high-risk fields such as healthcare and power systems.

To tackle this challenge, the paper proposes a novel offline federated policy optimization algorithm called DRPO (Doubly Regularized Federated Offline Policy Optimization). DRPO allows distributed agents to collaboratively learn decision-making policies solely from private static data without further interaction with the environment. By introducing a doubly regularized mechanism, DRPO effectively addresses the two-level distribution shift problem in offline FRL and ensures strict policy improvement in each federated learning round. Experimental results demonstrate that DRPO significantly outperforms baseline methods in standard offline reinforcement learning benchmarks.