Abstract:Reinforcement Learning (RL), recognized as an efficient learning approach, has achieved remarkable success across multiple fields and applications, including gaming, robotics, and autonomous vehicles. Classical single-agent reinforcement learning grapples with the imbalance of exploration and exploitation as well as limited generalization abilities. This methodology frequently leads to algorithms settling for suboptimal solutions that are tailored only to specific datasets. In this work, our aim is to train agent with efficient learning by decoupling exploration and utilization, so that agent can escaping the conundrum of suboptimal Solutions. In reinforcement learning, the previously imposed pessimistic punitive measures have deprived the model of its exploratory potential, resulting in diminished exploration capabilities. To address this, we have introduced an additional optimistic Actor to enhance the model's exploration ability, while employing a more constrained pessimistic Actor for performance evaluation. The above idea is implemented in the proposed OPARL (Optimistic and Pessimistic Actor Reinforcement Learning) algorithm. This unique amalgamation within the reinforcement learning paradigm fosters a more balanced and efficient approach. It facilitates the optimization of policies that concentrate on high-reward actions via pessimistic exploitation strategies while concurrently ensuring extensive state coverage through optimistic exploration. Empirical and theoretical investigations demonstrate that OPARL enhances agent capabilities in both utilization and exploration. In the most tasks of DMControl benchmark and Mujoco environment, OPARL performed better than state-of-the-art methods. Our code has released on

Preference-Guided Reinforcement Learning for Efficient Exploration

Beyond Reward: Offline Preference-guided Policy Optimization

STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs

Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation.

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

Provable Offline Preference-Based Reinforcement Learning

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

A Preference-based Reinforcement Learning Approach Using Reward Exploration for Decision Making

Trajectory-Oriented Policy Optimization with Sparse Rewards

UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systems

Hindsight Preference Learning for Offline Preference-based Reinforcement Learning

Provable Reward-Agnostic Preference-Based Reinforcement Learning

Online Policy Learning from Offline Preferences

Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation

Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

Reward-Free Exploration for Reinforcement Learning

Efficient Reinforcement Learning via Decoupling Exploration and Utilization

Reinforcing Long-Term Performance in Recommender Systems with User-Oriented Exploration Policy

Provably Efficient Exploration in Policy Optimization

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations