Abstract:Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30\% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.

Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update

Sample-Efficient Deep Reinforcement Learning Via Balance Sample

Sample Efficient Reinforcement Learning Method Via High Efficient Episodic Memory.

Episodic Reinforcement Learning with Expanded State-reward Space

Episodic Memory Deep Q-Networks

A Novel Adaptive Sampling Strategy for Deep Reinforcement Learning.

Episodic Reinforcement Learning with Associative Memory.

Efficient Reinforcement-Learning Control Algorithm Using Experience Reuse

Sample-efficient Deep Reinforcement Learning with Directed Associative Graph

Brain Inspired Episodic Memory Deep Q-Networks for Sparse Reward

Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning

State Distribution-Aware Sampling for Deep Q-Learning

Deep Q-Learning with Prioritized Sampling.

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Posterior Sampling for Deep Reinforcement Learning

Deep Deterministic Policy Gradient with Episode Experience Replay

Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments

Deep Reinforcement Learning with Parametric Episodic Memory

Sample Efficient Deep Reinforcement Learning via Local Planning

"Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations

Never Give Up: Learning Directed Exploration Strategies