Abstract:With the fast improvement of machine learning, reinforcement learning (RL) has been used to automate human tasks in different areas. However, training such agents is difficult and restricted to expert users. Moreover, it is mostly limited to simulation environments due to the high cost and safety concerns of interactions in the real world. Demonstration Learning is a paradigm in which an agent learns to perform a task by imitating the behavior of an expert shown in demonstrations. It is a relatively recent area in machine learning, but it is gaining significant traction due to having tremendous potential for learning complex behaviors from demonstrations. Learning from demonstration accelerates the learning process by improving sample efficiency, while also reducing the effort of the programmer. Due to learning without interacting with the environment, demonstration learning would allow the automation of a wide range of real world applications such as robotics and healthcare. This paper provides a survey of demonstration learning, where we formally introduce the demonstration problem along with its main challenges and provide a comprehensive overview of the process of learning from demonstrations from the creation of the demonstration data set, to learning methods from demonstrations, and optimization by combining demonstration learning with different machine learning methods. We also review the existing benchmarks and identify their strengths and limitations. Additionally, we discuss the advantages and disadvantages of the paradigm as well as its main applications. Lastly, we discuss our perspective on open problems and research directions for this rapidly growing field.

What problem does this paper attempt to address?

The paper primarily explores the field of Demonstration Learning, a branch of machine learning where agents learn to perform tasks by imitating the behavior demonstrated by experts. The goal of the paper is to provide a comprehensive review of Demonstration Learning, covering methods from creating demonstration datasets to utilizing these data for learning, as well as optimizing the process by combining different machine learning techniques. Specifically, the paper first defines the problem of Demonstration Learning and introduces related concepts such as the Markov Decision Process (MDP) in reinforcement learning. Next, the paper discusses various methods and techniques for collecting demonstration data, including direct demonstrations (e.g., teleoperation) and indirect demonstrations (e.g., observation). The paper also analyzes how to process demonstration data to make it useful for learning agents, which may involve mapping the state-action pairs demonstrated by the teacher into a form that the learning agent can understand. Furthermore, the paper outlines various methods for learning from demonstration data, including Behavior Cloning, Inverse Reinforcement Learning (IRL), and offline reinforcement learning. These methods aim to help agents learn strategies from demonstration data, enabling them to perform complex tasks while reducing the trial-and-error required during the exploration phase. The paper also reviews existing benchmarks and discusses the advantages and limitations of Demonstration Learning, as well as future research directions in this field. For example, one significant advantage of Demonstration Learning is that it can accelerate the learning process by improving sample efficiency, and since it does not require interaction with the environment for learning, it has broad application prospects in fields such as robotics and healthcare. However, it also faces challenges, such as the need for high-quality demonstration datasets to cover all possible state spaces and how to handle distribution shifts in the dataset. In summary, this paper provides an exhaustive review of the field of Demonstration Learning, introducing current research progress and pointing out future research directions. It serves as a valuable resource for researchers who wish to gain an in-depth understanding of Demonstration Learning and its applications.

A Survey of Demonstration Learning

Recent Advances in Robot Learning from Demonstration

A survey of robot learning from demonstration

Demonstration Guided Actor-Critic Deep Reinforcement Learning for Fast Teaching of Robots in Dynamic Environments

Active Deep Q-learning with Demonstration

Boosting Reinforcement Learning and Planning with Demonstrations: A Survey

A reinforcement learning algorithm acquires demonstration from the training agent by dividing the task space

Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

Demonstration actor critic

Learning from Successful and Failed Demonstrations via Optimization

Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration

Residual Reinforcement Learning from Demonstrations

Deep Q-learning From Demonstrations

Learning Traffic Signal Control from Demonstrations

Overcoming Exploration in Reinforcement Learning with Demonstrations

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Extended Reality System for Robotic Learning from Human Demonstration

Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning

How Can Everyday Users Efficiently Teach Robots by Demonstrations?

A Framework for Learning from Demonstration with Minimal Human Effort

Learning from Demonstration Framework for Multi-Robot Systems Using Interaction Keypoints and Soft Actor-Critic Methods