Abstract:Exploration strategies in continuous action space are often heuristic due to the infinite actions, and these kinds of methods cannot derive a general conclusion. In prior work, it has been shown that policy-based exploration is beneficial for continuous action space in deterministic policy reinforcement learning(DPRL). However, policy-based exploration in DPRL has two prominent issues: aimless exploration and policy divergence, and the policy gradient for exploration is only sometimes helpful due to inaccurate estimation. Based on the double-Q function framework, we introduce a novel exploration strategy to mitigate these issues, separate from the policy gradient. We first propose the greedy Q softmax update schema for Q value update. The expected Q value is derived by weighted summing the conservative Q value over actions, and the weight is the corresponding greedy Q value. Greedy Q takes the maximum value of the two Q functions, and conservative Q takes the minimum value of the two different Q functions. For practicality, this theoretical basis is then extended to allow us to combine action exploration with the Q value update, except for the premise that we have a surrogate policy that behaves like this exploration policy. In practice, we construct such an exploration policy with a few sampled actions, and to meet the premise, we learn such a surrogate policy by minimizing the KL divergence between the target policy and the exploration policy constructed by the conservative Q. We evaluate our method on the Mujoco benchmark and demonstrate superior performance compared to previous state-of-the-art methods across various environments, particularly in the most complex Humanoid environment.

Leveraging exploration in off-policy algorithms via normalizing flows

A Scalable Derivative-free Exploration Approach for Reinforcement Learning

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

Exploiting the Sign of the Advantage Function to Learn Deterministic Policies in Continuous Domains

Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience

Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning

Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies

FlowPG: Action-constrained Policy Gradient with Normalizing Flows

Reducing Entropy Overestimation in Soft Actor Critic Using Dual Policy Network

Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

Provably Correct Optimization and Exploration with Non-linear Policies

Unified Policy Optimization for Continuous-action Reinforcement Learning in Non-stationary Tasks and Games

Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluation

Improving exploration in policy gradient search: Application to symbolic optimization

Promoting Stochasticity for Expressive Policies Via a Simple and Efficient Regularization Method.

Careful at Estimation and Bold at Exploration

Generalizing soft actor-critic algorithms to discrete action spaces

Off-Policy Deep Reinforcement Learning with Analogous Disentangled Exploration

Exploration in Feature Space for Reinforcement Learning