Abstract:Policy gradient methods in actor-critic reinforcement learning (RL) have become perhaps the most promising approaches to solving continuous optimal control problems. However, the trial-and-error nature of RL and the inherent randomness associated with solution approximations cause variations in the learned optimal values and policies. This has significantly hindered their successful deployment in real life applications where control responses need to meet dynamic performance criteria deterministically. Here we propose a novel phased actor in actor-critic (PAAC) method, aiming at improving policy gradient estimation and thus the quality of the control policy. Specifically, PAAC accounts for both $Q$ value and TD error in its actor update. We prove qualitative properties of PAAC for learning convergence of the value and policy, solution optimality, and stability of system dynamics. Additionally, we show variance reduction in policy gradient estimation. PAAC performance is systematically and quantitatively evaluated in this study using DeepMind Control Suite (DMC). Results show that PAAC leads to significant performance improvement measured by total cost, learning variance, robustness, learning speed and success rate. As PAAC can be piggybacked onto general policy gradient learning frameworks, we select well-known methods such as direct heuristic dynamic programming (dHDP), deep deterministic policy gradient (DDPG) and their variants to demonstrate the effectiveness of PAAC. Consequently we provide a unified view on these related policy gradient algorithms.

Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learning

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning

A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correction

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Multi-task Actor-Critic with Knowledge Transfer via a Shared Critic.

Multi-Task Policy Search

AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers

DiGrad: Multi-Task Reinforcement Learning with Shared Actions

Latent-Conditioned Policy Gradient for Multi-Objective Deep Reinforcement Learning

Theoretical Study of Conflict-Avoidant Multi-Objective Reinforcement Learning

Policy Consensus-Based Distributed Deterministic Multi-Agent Reinforcement Learning over Directed Graphs

Decentralized Multi-Task Reinforcement Learning Policy Gradient Method with Momentum over Networks.

Actor-Critic Reinforcement Learning with Phased Actor

On the Convergence Theory of Meta Reinforcement Learning with Personalized Policies

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective

A Brain-Inspired Incremental Multi-task Reinforcement Learning Approach