Abstract:Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30\% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.

Self-Guided Actor-Critic: Reinforcement Learning from Adaptive Expert Demonstrations

Reinforcement Learning from Demonstrations by Novel Interactive Expert and Application to Automatic Berthing Control Systems for Unmanned Surface Vessel

Self-play Reinforcement Learning with Comprehensive Critic in Computer Games

Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control

Demonstration actor critic

A Deep Reinforcement Learning Algorithm with Expert Demonstrations and Supervised Loss and Its Application in Autonomous Driving

GRAC: Self-Guided and Self-Regularized Actor-Critic

Demonstration Guided Actor-Critic Deep Reinforcement Learning for Fast Teaching of Robots in Dynamic Environments

Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations

Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration

Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance

GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback

DSAC-T: Distributional Soft Actor-Critic with Three Refinements

LIDAR: Learning from Imperfect Demonstrations with Advantage Rectification

Tracking Control of AUV Via Novel Soft Actor-Critic and Suboptimal Demonstrations

Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble

"Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations

Explorer-Actor-Critic: Better Actors for Deep Reinforcement Learning

Active Deep Q-learning with Demonstration

Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving

Skill-Enhanced Reinforcement Learning Acceleration from Demonstrations