Abstract:Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30\% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.

WayEx: Waypoint Exploration using a Single Demonstration

Expert demonstrations guide reward decomposition for multi-agent cooperation

Generalize Robot Learning from Demonstration to Variant Scenarios with Evolutionary Policy Gradient

Overcoming Exploration in Reinforcement Learning with Demonstrations

Guided Exploration with Proximal Policy Optimization using a Single Demonstration

Learning to Solve Tasks with Exploring Prior Behaviours

VIEW: Visual Imitation Learning with Waypoints

Efficient Hindsight Reinforcement Learning Using Demonstrations for Robotic Tasks with Sparse Rewards

Waypoint-Based Imitation Learning for Robotic Manipulation

Extending Policy from One-Shot Learning through Coaching

Adaptive Cooperative Exploration for Reinforcement Learning from Imperfect Demonstrations

Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations

Show me the Way: Intrinsic Motivation from Demonstrations

Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback

Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

"Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations

Policy Gradient from Demonstration and Curiosity

Enabling Robots to Identify Missing Steps in Robot Tasks for Guided Learning from Demonstration

A reinforcement learning algorithm acquires demonstration from the training agent by dividing the task space

Reinforcement learning from expert demonstrations with application to redundant robot control