Abstract:Exploration strategies in continuous action space are often heuristic due to the infinite actions, and these kinds of methods cannot derive a general conclusion. In prior work, it has been shown that policy-based exploration is beneficial for continuous action space in deterministic policy reinforcement learning(DPRL). However, policy-based exploration in DPRL has two prominent issues: aimless exploration and policy divergence, and the policy gradient for exploration is only sometimes helpful due to inaccurate estimation. Based on the double-Q function framework, we introduce a novel exploration strategy to mitigate these issues, separate from the policy gradient. We first propose the greedy Q softmax update schema for Q value update. The expected Q value is derived by weighted summing the conservative Q value over actions, and the weight is the corresponding greedy Q value. Greedy Q takes the maximum value of the two Q functions, and conservative Q takes the minimum value of the two different Q functions. For practicality, this theoretical basis is then extended to allow us to combine action exploration with the Q value update, except for the premise that we have a surrogate policy that behaves like this exploration policy. In practice, we construct such an exploration policy with a few sampled actions, and to meet the premise, we learn such a surrogate policy by minimizing the KL divergence between the target policy and the exploration policy constructed by the conservative Q. We evaluate our method on the Mujoco benchmark and demonstrate superior performance compared to previous state-of-the-art methods across various environments, particularly in the most complex Humanoid environment.

Efficient and Stable Information Directed Exploration for Continuous Reinforcement Learning

Adaptive Exploration Network Policy for Effective Exploration in Reinforcement Learning

Centralized Cooperative Exploration Policy for Continuous Control Tasks

Information-Directed Exploration for Deep Reinforcement Learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

Careful at Estimation and Bold at Exploration

Deep intrinsically motivated exploration in continuous control

Active Exploration Deep Reinforcement Learning for Continuous Action Space with Forward Prediction

Never Give Up: Learning Directed Exploration Strategies

Guided Exploration in Reinforcement Learning via Monte Carlo Critic Optimization

A Study of Qualitative Knowledge-Based Exploration for Continuous Deep Reinforcement Learning.

Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction

Efficient Exploration in Continuous-time Model-based Reinforcement Learning

Efficient and Scalable Exploration Via Estimation-Error

Off-Policy Deep Reinforcement Learning with Analogous Disentangled Exploration

Self-supervised Sequential Information Bottleneck for Robust Exploration in Deep Reinforcement Learning

The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective

Optimized Feature Extraction for Sample Efficient Deep Reinforcement Learning

An Efficient Robot Exploration Method Based on Heuristics Biased Sampling

Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration