Abstract:Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30\% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.

Policy Gradient from Demonstration and Curiosity

Generalize Robot Learning from Demonstration to Variant Scenarios with Evolutionary Policy Gradient

Expert demonstrations guide reward decomposition for multi-agent cooperation

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

Guided Exploration with Proximal Policy Optimization using a Single Demonstration

Overcoming Exploration in Reinforcement Learning with Demonstrations

Trajectory-Oriented Policy Optimization with Sparse Rewards

Behind the Myth of Exploration in Policy Gradients

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration

Model-free Policy Learning with Reward Gradients

Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

Learning Diverse Policies with Soft Self-Generated Guidance

Curious Explorer: a Provable Exploration Strategy in Policy Learning

Demonstration actor critic

Adaptive Cooperative Exploration for Reinforcement Learning from Imperfect Demonstrations

Efficient sample reuse in policy gradients with parameter-based exploration

A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments

"Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations

Policy Augmentation: An Exploration Strategy for Faster Convergence of Deep Reinforcement Learning Algorithms