Abstract:In this paper, we propose actor-director-critic, a new framework for deep reinforcement learning. Compared with the actor-critic framework, the director role is added, and action classification and action evaluation are applied simultaneously to improve the decision-making performance of the agent. Firstly, the actions of the agent are divided into high quality actions and low quality actions according to the rewards returned from the environment. Then, the director network is trained to have the ability to discriminate high and low quality actions and guide the actor network to reduce the repetitive exploration of low quality actions in the early stage of training. In addition, we propose an improved double estimator method to better solve the problem of overestimation in the field of reinforcement learning. For the two critic networks used, we design two target critic networks for each critic network instead of one. In this way, the target value of each critic network can be calculated by taking the average of the outputs of the two target critic networks, which is more stable and accurate than using only one target critic network to obtain the target value. In order to verify the performance of the actor-director-critic framework and the improved double estimator method, we applied them to the TD3 algorithm to improve the TD3 algorithm. Then, we carried out experiments in multiple environments in MuJoCo and compared the experimental data before and after the algorithm improvement. The final experimental results show that the improved algorithm can achieve faster convergence speed and higher total return.

Overcoming Delayed Feedback in Reinforcement Learning Using Actor Ensembles

Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments

Actor-Critic Reinforcement Learning with Phased Actor

Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning

Overcoming Delayed Feedback Via Overlook Decision Making

DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays

TD3 with Composite Forgetting Prioritized Experience Replay

Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays

Actor Prioritized Experience Replay

Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning

Optimizing TD3 for 7-DOF Robotic Arm Grasping: Overcoming Suboptimality with Exploration-Enhanced Contrastive Learning

Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control

Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations

Efficient Continuous Control with Double Actors and Regularized Critics

Robust Adaptive Ensemble Adversary Reinforcement Learning

Continuous Reinforcement Learning From Human Demonstrations With Integrated Experience Replay For Autonomous Driving

Actor-Director-Critic: A Novel Deep Reinforcement Learning Framework

Policy ensemble gradient for continuous control problems in deep reinforcement learning

Twin-Delayed Ddpg: A Deep Reinforcement Learning Technique To Model A Continuous Movement Of An Intelligent Robot Agent

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Addressing Delays in Reinforcement Learning Via Delayed Adversarial Imitation Learning