Abstract:The sparsity of reward feedback remains a challenging problem in online deep reinforcement learning (DRL). Previous approaches have utilized offline demonstrations to achieve impressive results in multiple hard tasks. However, these approaches place high demands on demonstration quality, and obtaining expert-like actions is often costly and unrealistic. To tackle these problems, we propose a simple and efficient algorithm called Policy Optimization with Smooth Guidance (POSG), which leverages a small set of state-only demonstrations (where expert action information is not included in demonstrations) to indirectly make approximate and feasible long-term credit assignments and facilitate exploration. Specifically, we first design a trajectory-importance evaluation mechanism to determine the quality of the current trajectory against demonstrations. Then, we introduce a guidance reward computation technology based on trajectory importance to measure the impact of each state-action pair, fusing the demonstrator's state distribution with reward information into the guidance reward. We theoretically analyze the performance improvement caused by smooth guidance rewards and derive a new worst-case lower bound on the performance improvement. Extensive results demonstrate POSG's significant advantages in control performance and convergence speed in four sparse-reward environments, including the grid-world maze, Hopper-v4, HalfCheetah-v4, and Ant maze. Notably, the specific metrics and quantifiable results are investigated to demonstrate the superiority of POSG.

Demonstration-Based Proximal Policy Optimization with Action Guidance

Behavior Proximal Policy Optimization

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

Guided Exploration with Proximal Policy Optimization using a Single Demonstration

Proximal policy optimization via enhanced exploration efficiency

Proximal Policy Optimization Based on Self-directed Action Selection

Proximal Policy Optimization with Future Rewards

Trust Region-Guided Proximal Policy Optimization

Proximal Policy Optimization with Policy Feedback

Proximal Policy Optimization with Adaptive Exploration

Proximal Policy Optimization Algorithms

Trajectory-Oriented Policy Optimization with Sparse Rewards

Fast-PPO: Proximal Policy Optimization with Optimal Baseline Method

Provably Efficient Exploration in Policy Optimization

An Improved Proximal Policy Optimization Algorithm for Autonomous Driving Decision-Making

Anomaly Guided Policy Learning from Imperfect Demonstrations.

Truly Proximal Policy Optimization

Demonstration actor critic

Policy Optimization with Model-based Explorations

Beyond the Boundaries of Proximal Policy Optimization

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards