Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Generative Model

Akira Kinose,T. Taniguchi

Abstract:,

Computer Science,Engineering

What problem does this paper attempt to address?

Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model

Akira Kinose,Tadahiro Taniguchi

DOI: https://doi.org/10.1080/01691864.2020.1778521

2019-10-16

Abstract:Integration of reinforcement learning and imitation learning is an important problem that has been studied for a long time in the field of intelligent robotics. Reinforcement learning optimizes policies to maximize the cumulative reward, whereas imitation learning attempts to extract general knowledge about the trajectories demonstrated by experts, i.e., demonstrators. Because each of them has their own drawbacks, methods combining them and compensating for each set of drawbacks have been explored thus far. However, many of the methods are heuristic and do not have a solid theoretical basis. In this paper, we present a new theory for integrating reinforcement and imitation learning by extending the probabilistic generative model framework for reinforcement learning, {\it plan by inference}. We develop a new probabilistic graphical model for reinforcement learning with multiple types of rewards and a probabilistic graphical model for Markov decision processes with multiple optimality emissions (pMDP-MO). Furthermore, we demonstrate that the integrated learning method of reinforcement learning and imitation learning can be formulated as a probabilistic inference of policies on pMDP-MO by considering the output of the discriminator in generative adversarial imitation learning as an additional optimal emission observation. We adapt the generative adversarial imitation learning and task-achievement reward to our proposed framework, achieving significantly better performance than agents trained with reinforcement learning or imitation learning alone. Experiments demonstrate that our framework successfully integrates imitation and reinforcement learning even when the number of demonstrators is only a few.

Machine Learning,Artificial Intelligence
Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning

Kyoichiro Kobayashi,Takato Horii,Ryo Iwaki,Yukie Nagai,Minoru Asada

DOI: https://doi.org/10.48550/arXiv.1911.00238

IF: 5.414

2019-11-01

Machine Learning

Abstract:Generative adversarial imitation learning (GAIL) has attracted increasing attention in the field of robot learning. It enables robots to learn a policy to achieve a task demonstrated by an expert while simultaneously estimating the reward function behind the expert's behaviors. However, this framework is limited to learning a single task with a single reward function. This study proposes an extended framework called situated GAIL (S-GAIL), in which a task variable is introduced to both the discriminator and generator of the GAIL framework. The task variable has the roles of discriminating different contexts and making the framework learn different reward functions and policies for multiple tasks. To achieve the early convergence of learning and robustness during reward estimation, we introduce a term to adjust the entropy regularization coefficient in the generator's objective function. Our experiments using two setups (navigation in a discrete grid world and arm reaching in a continuous space) demonstrate that the proposed framework can acquire multiple reward functions and policies more effectively than existing frameworks. The task variable enables our framework to differentiate contexts while sharing common knowledge among multiple tasks.
On Computation and Generalization of Generative Adversarial Imitation Learning.

Minshuo Chen,Yizhou Wang,Tianyi Liu,Zhuoran Yang,Xingguo Li,Zhaoran Wang,Tuo Zhao

DOI: https://doi.org/10.48550/arxiv.2001.02792

2020-01-01

Abstract:Generative Adversarial Imitation Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies. Different from Reinforcement Learning (RL), GAIL takes advantage of demonstration data by experts (e.g., human), and learns both the policy and reward function of the unknown environment. Despite the significant empirical progresses, the theory behind GAIL is still largely unknown. The major difficulty comes from the underlying temporal dependency of the demonstration data and the minimax computational formulation of GAIL without convex-concave structure. To bridge such a gap between theory and practice, this paper investigates the theoretical properties of GAIL. Specifically, we show: (1) For GAIL with general reward parameterization, the generalization can be guaranteed as long as the class of the reward functions is properly controlled; (2) For GAIL, where the reward is parameterized as a reproducing kernel function, GAIL can be efficiently solved by stochastic first order optimization algorithms, which attain sublinear convergence to a stationary solution. To the best of our knowledge, these are the first results on statistical and computational guarantees of imitation learning with reward/policy function approximation. Numerical experiments are provided to support our analysis.
Acgail: Imitation Learning About Multiple Intentions With Auxiliary Classifier Gans

Jiahao Lin,Zongzhang Zhang

DOI: https://doi.org/10.1007/978-3-319-97304-3_25

2018-01-01

Abstract:As an important solution to decision-making problems, imitation learning learns expert behavior from example demonstrations provided by experts, without the necessity of a predefined reward function as in reinforcement learning. Traditionally, imitation learning assumes that demonstrations are generated from single latent expert intention. One promising method in this line is generative adversarial imitation learning (GAIL), designed to work in large environments. It can be thought as a model-free imitation learning built on top of generative adversarial networks (GANs). However, GAIL fails to learn well when handling expert demonstrations under multiple intentions, which can be labeled by latent intentions. In this paper, we propose to add an auxiliary classifier model to GAIL, from which we derive a novel variant of GAIL, named ACGAIL, allowing label conditioning in imitation learning about multiple intentions. Experimental results on several MuJoCo tasks indicate that ACGAIL can achieve significant performance improvements over existing methods, e.g., GAIL and InfoGAIL, when dealing with label-conditional imitation learning about multiple intentions.
Generative Adversarial Imitation Learning from Failed Experiences

Jiacheng Zhu,Jiahao Lin,Meng Wang,Yingfeng Chen,Changjie Fan,Chong Jiang,Zongzhang Zhang

DOI: https://doi.org/10.1609/AAAI.V34I10.7271

2020-01-01

Abstract:Imitation learning provides a family of promising methods that learn policies from expert demonstrations directly. As a model-free and on-line imitation learning method, generative adversarial imitation learning (GAIL) generalizes well to unseen situations and can handle complex problems. In this paper, we propose a novel variant of GAIL called GAIL from failed experiences (GAILFE). GAILFE allows an agent to utilize failed experiences in the training process. Moreover, a constrained optimization objective is formalized in GAILFE to balance learning from given demonstrations and from self-generated failed experiences. Empirically, compared with GAIL, GAILFE can improve sample efficiency and learning speed over different tasks.
GAILPG: Multi-Agent Policy Gradient with Generative Adversarial Imitation Learning

Wei Li,Shiyi Huang,Ziming Qiu,Aiguo Song

DOI: https://doi.org/10.1109/tg.2024.3375515

IF: 1.237

2024-01-01

IEEE Transactions on Games

Abstract:In reinforcement learning, the agents need to sufficiently explore the environment and efficiently exploit the existing experiences before finding the solution to the tasks, particularly in cooperative multi-agent scenarios where the state and action spaces grow exponentially with the number of agents. Hence, enhancing the exploration ability of agents and improving the utilization efficiency of experiences are two critical issues in cooperative multi-agent reinforcement learning. We propose a novel method called Generative Adversarial Imitation Learning Policy Gradients (GAILPG). The contributions of GAILPG are as follows: (a) we integrate generative adversarial self-imitation learning into the multi-agent actor-critic framework to improve the utilization efficiency of experiences, thus further assisting the policy training; (b) we design a new curiosity module to enhance the exploration ability of the agents. Experimental results on the StarCraft II micromanagement benchmark demonstrate that GAILPG surpasses state-of-the-art policy-based methods and is even on par with the value-based methods. And the ablation experiments validate the reasonability of the discriminator module and the curiosity module encapsulated in our method.

computer science, artificial intelligence, software engineering
A Pragmatic Look at Deep Imitation Learning

Kai Arulkumaran,Dan Ogawa Lillrank

2023-09-20

Abstract:The introduction of the generative adversarial imitation learning (GAIL) algorithm has spurred the development of scalable imitation learning approaches using deep neural networks. Many of the algorithms that followed used a similar procedure, combining on-policy actor-critic algorithms with inverse reinforcement learning. More recently there have been an even larger breadth of approaches, most of which use off-policy algorithms. However, with the breadth of algorithms, everything from datasets to base reinforcement learning algorithms to evaluation settings can vary, making it difficult to fairly compare them. In this work we re-implement 6 different IL algorithms, updating 3 of them to be off-policy, base them on a common off-policy algorithm (SAC), and evaluate them on a widely-used expert trajectory dataset (D4RL) for the most common benchmark (MuJoCo). After giving all algorithms the same hyperparameter optimisation budget, we compare their results for a range of expert trajectories. In summary, GAIL, with all of its improvements, consistently performs well across a range of sample sizes, AdRIL is a simple contender that performs well with one important hyperparameter to tune, and behavioural cloning remains a strong baseline when data is more plentiful.

Machine Learning,Neural and Evolutionary Computing
Distributional generative adversarial imitation learning with reproducing kernel generalization

Yirui Zhou,Mengxiao Lu,Xiaowei Liu,Zhengping Che,Zhiyuan Xu,Jian Tang,Yangchun Zhang,Yan Peng,Yaxin Peng

DOI: https://doi.org/10.1016/j.neunet.2023.05.027

Abstract:Generative adversarial imitation learning (GAIL) regards imitation learning (IL) as a distribution matching problem between the state-action distributions of the expert policy and the learned policy. In this paper, we focus on the generalization and computational properties of policy classes. We prove that the generalization can be guaranteed in GAIL when the class of policies is well controlled. With the capability of policy generalization, we introduce distributional reinforcement learning (RL) into GAIL and propose the greedy distributional soft gradient (GDSG) algorithm to solve GAIL. The main advantages of GDSG can be summarized as: (1) Q-value overestimation, a crucial factor leading to the instability of GAIL with off-policy training, can be alleviated by distributional RL. (2) By considering the maximum entropy objective, the policy can be improved in terms of performance and sample efficiency through sufficient exploration. Moreover, GDSG attains a sublinear convergence rate to a stationary solution. Comprehensive experimental verification in MuJoCo environments shows that GDSG can mimic expert demonstrations better than previous GAIL variants.
Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulation

Yoshihisa Tsurumine,Takamitsu Matsubara

DOI: https://doi.org/10.48550/arXiv.2209.10149

2022-09-21

Abstract:Generative Adversarial Imitation Learning (GAIL) can learn policies without explicitly defining the reward function from demonstrations. GAIL has the potential to learn policies with high-dimensional observations as input, e.g., images. By applying GAIL to a real robot, perhaps robot policies can be obtained for daily activities like washing, folding clothes, cooking, and cleaning. However, human demonstration data are often imperfect due to mistakes, which degrade the performance of the resulting policies. We address this issue by focusing on the following features: 1) many robotic tasks are goal-reaching tasks, and 2) labeling such goal states in demonstration data is relatively easy. With these in mind, this paper proposes Goal-Aware Generative Adversarial Imitation Learning (GA-GAIL), which trains a policy by introducing a second discriminator to distinguish the goal state in parallel with the first discriminator that indicates the demonstration data. This extends a standard GAIL framework to more robustly learn desirable policies even from imperfect demonstrations through a goal-state discriminator that promotes achieving the goal state. Furthermore, GA-GAIL employs the Entropy-maximizing Deep P-Network (EDPN) as a generator, which considers both the smoothness and causal entropy in the policy update, to achieve stable policy learning from two discriminators. Our proposed method was successfully applied to two real-robotic cloth-manipulation tasks: turning a handkerchief over and folding clothes. We confirmed that it learns cloth-manipulation policies without task-specific reward function design. Video of the real experiments are available at <a class="link-external link-https" href="https://youtu.be/h_nII2ooUrE" rel="external noopener nofollow">this https URL</a>.

Robotics
Addressing reward bias in Adversarial Imitation Learning with neutral reward functions

Rohit Jena,Siddharth Agrawal,Katia Sycara

DOI: https://doi.org/10.48550/arXiv.2009.09467

2020-09-21

Abstract:Generative Adversarial Imitation Learning suffers from the fundamental problem of reward bias stemming from the choice of reward functions used in the algorithm. Different types of biases also affect different types of environments - which are broadly divided into survival and task-based environments. We provide a theoretical sketch of why existing reward functions would fail in imitation learning scenarios in task based environments with multiple terminal states. We also propose a new reward function for GAIL which outperforms existing GAIL methods on task based environments with single and multiple terminal states and effectively overcomes both survival and termination bias.

Machine Learning,Robotics
CRIL: Continual Robot Imitation Learning via Generative and Prediction Model

Chongkai Gao,Haichuan Gao,Shangqi Guo,Tianren Zhang,Feng Chen

DOI: https://doi.org/10.1109/iros51168.2021.9636069

2021-09-27

Abstract:Imitation learning (IL) algorithms have shown promising results for robots to learn skills from expert demonstrations. However, they need multi-task demonstrations to be provided at once for acquiring diverse skills, which is difficult in real world. In this work we study how to realize continual imitation learning ability that empowers robots to continually learn new tasks one by one, thus reducing the burden of multitask IL and accelerating the process of new task learning at the same time. We propose a novel trajectory generation model that employs both a generative adversarial network and a dynamics-aware prediction model to generate pseudo trajectories from all learned tasks in the new task learning process. Our experiments on both simulation and real-world manipulation tasks demonstrate the effectiveness of our method.
C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

Tianjiao Luo,Tim Pearce,Huayu Chen,Jianfei Chen,Jun Zhu

2024-10-29

Abstract:Generative Adversarial Imitation Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. A major drawback of GAIL is its training instability - it inherits the complex training dynamics of GANs, and the distribution shift introduced by RL. This can cause oscillations during training, harming its sample efficiency and final policy performance. Recent work has shown that control theory can help with the convergence of a GAN's training. This paper extends this line of work, conducting a control-theoretic analysis of GAIL and deriving a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting. Based on this, we propose a practical algorithm 'Controlled-GAIL' (C-GAIL). On MuJoCo tasks, our controlled variant is able to speed up the rate of convergence, reduce the range of oscillation and match the expert's distribution more closely both for vanilla GAIL and GAIL-DAC.

Machine Learning,Systems and Control
Interpretable Generative Adversarial Imitation Learning

Wenliang Liu,Danyang Li,Erfan Aasi,Roberto Tron,Calin Belta

2024-02-16

Abstract:Imitation learning methods have demonstrated considerable success in teaching autonomous systems complex tasks through expert demonstrations. However, a limitation of these methods is their lack of interpretability, particularly in understanding the specific task the learning agent aims to accomplish. In this paper, we propose a novel imitation learning method that combines Signal Temporal Logic (STL) inference and control synthesis, enabling the explicit representation of the task as an STL formula. This approach not only provides a clear understanding of the task but also allows for the incorporation of human knowledge and adaptation to new scenarios through manual adjustments of the STL formulae. Additionally, we employ a Generative Adversarial Network (GAN)-inspired training approach for both the inference and the control policy, effectively narrowing the gap between the expert and learned policies. The effectiveness of our algorithm is demonstrated through two case studies, showcasing its practical applicability and adaptability.

Machine Learning,Systems and Control
Generative predecessor models for sample-efficient imitation learning

Yannick Schroecker,Mel Vecerik,Jonathan Scholz

DOI: https://doi.org/10.48550/arXiv.1904.01139

IF: 5.414

2019-04-01

Machine Learning

Abstract:We propose Generative Predecessor Models for Imitation Learning (GPRIL), a novel imitation learning algorithm that matches the state-action distribution to the distribution observed in expert demonstrations, using generative models to reason probabilistically about alternative histories of demonstrated states. We show that this approach allows an agent to learn robust policies using only a small number of expert demonstrations and self-supervised interactions with the environment. We derive this approach from first principles and compare it empirically to a state-of-the-art imitation learning method, showing that it outperforms or matches its performance on two simulated robot manipulation tasks and demonstrate significantly higher sample efficiency by applying the algorithm on a real robot.
Diffusion-Reward Adversarial Imitation Learning

Chun-Mao Lai,Hsiang-Chun Wang,Ping-Chun Hsieh,Yu-Chiang Frank Wang,Min-Hung Chen,Shao-Hua Sun

2024-10-23

Abstract:Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. Generative adversarial imitation learning (GAIL) formulates imitation learning as adversarial learning, employing a generator policy learning to imitate expert behaviors and discriminator learning to distinguish the expert demonstrations from agent trajectories. Despite its encouraging results, GAIL training is often brittle and unstable. Inspired by the recent dominance of diffusion models in generative modeling, we propose Diffusion-Reward Adversarial Imitation Learning (DRAIL), which integrates a diffusion model into GAIL, aiming to yield more robust and smoother rewards for policy learning. Specifically, we propose a diffusion discriminative classifier to construct an enhanced discriminator, and design diffusion rewards based on the classifier's output for policy learning. Extensive experiments are conducted in navigation, manipulation, and locomotion, verifying DRAIL's effectiveness compared to prior imitation learning methods. Moreover, additional experimental results demonstrate the generalizability and data efficiency of DRAIL. Visualized learned reward functions of GAIL and DRAIL suggest that DRAIL can produce more robust and smoother rewards. Project page: <a class="link-external link-https" href="https://nturobotlearninglab.github.io/DRAIL/" rel="external noopener nofollow">this https URL</a>

Machine Learning,Artificial Intelligence,Robotics
Self-imitation guided goal-conditioned reinforcement learning

Yao Li,YuHui Wang,XiaoYang Tan

DOI: https://doi.org/10.1016/j.patcog.2023.109845

IF: 8

2023-08-04

Pattern Recognition

Abstract:Goal-conditioned reinforcement learning (GCRL) aims to control agents to reach desired goals, which poses a significant challenge due to task-specific variations in configurations. However, current GCRL methods suffer from limitations in sample efficiency and the need for substantial training data. While existing self-imitation-based GCRL approaches can improve sample efficiency, their scalability to large-scale tasks is limited. In this paper, we propose integrating self-imitation learning with goal-conditioned RL methods into a compatible and reasonable framework. Specifically, we introduce a novel target action value function to aggregate self-imitation learning and goal-conditioned reinforcement learning. The designed target value effectively combines these two policy training mechanisms to accomplish specific tasks. Moreover, we theoretically demonstrate that our approach can learn a superior policy compared to both self-imitation learning and goal-conditioned reinforcement learning. Additionally, experimental results showcase the stability and effectiveness of our method compared to existing approaches in various challenging robotic control tasks.

computer science, artificial intelligence,engineering, electrical & electronic
Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets

Cong Fei,Bin Wang,Yuzheng Zhuang,Zongzhang Zhang,Jianye Hao,Hongbo Zhang,Xuewu Ji,Wulong Liu

DOI: https://doi.org/10.24963/ijcai.2020/405

2020-01-01

Abstract:Generative adversarial imitation learning (GAIL) has shown promising results by taking advantage of generative adversarial nets, especially in the field of robot learning. However, the requirement of isolated single modal demonstrations limits the scalability of the approach to real world scenarios such as autonomous vehicles' demand for a proper understanding of human drivers' behavior. In this paper, we propose a novel multi-modal GAIL framework, named Triple-GAIL, that is able to learn skill selection and imitation jointly from both expert demonstrations and continuously generated experiences with data augmentation purpose by introducing an auxiliary selector. We provide theoretical guarantees on the convergence to optima for both of the generator and the selector respectively. Experiments on real driver trajectories and real-time strategy game datasets demonstrate that Triple-GAIL can better fit multi-modal behaviors close to the demonstrators and outperforms state-of-the-art methods.
Latent Policies for Adversarial Imitation Learning

Tianyu Wang,Nikhil Karnwal,Nikolay Atanasov

DOI: https://doi.org/10.48550/arXiv.2206.11299

IF: 5.414

2022-06-22

Machine Learning

Abstract:This paper considers learning robot locomotion and manipulation tasks from expert demonstrations. Generative adversarial imitation learning (GAIL) trains a discriminator that distinguishes expert from agent transitions, and in turn use a reward defined by the discriminator output to optimize a policy generator for the agent. This generative adversarial training approach is very powerful but depends on a delicate balance between the discriminator and the generator training. In high-dimensional problems, the discriminator training may easily overfit or exploit associations with task-irrelevant features for transition classification. A key insight of this work is that performing imitation learning in a suitable latent task space makes the training process stable, even in challenging high-dimensional problems. We use an action encoder-decoder model to obtain a low-dimensional latent action space and train a LAtent Policy using Adversarial imitation Learning (LAPAL). The encoder-decoder model can be trained offline from state-action pairs to obtain a task-agnostic latent action representation or online, simultaneously with the discriminator and generator training, to obtain a task-aware latent action representation. We demonstrate that LAPAL training is stable, with near-monotonic performance improvement, and achieves expert performance in most locomotion and manipulation tasks, while a GAIL baseline converges slower and does not achieve expert performance in high-dimensional environments.
Accelerating Reinforcement Learning through Implicit Imitation

C. Boutilier,B. Price

DOI: https://doi.org/10.1613/jair.898

2011-06-03

Abstract:Imitation can be viewed as a means of enhancing learning in multiagent environments. It augments an agent's ability to learn useful behaviors by making intelligent use of the knowledge implicit in behaviors demonstrated by cooperative teachers or other more experienced agents. We propose and study a formal model of implicit imitation that can accelerate reinforcement learning dramatically in certain cases. Roughly, by observing a mentor, a reinforcement-learning agent can extract information about its own capabilities in, and the relative value of, unvisited parts of the state space. We study two specific instantiations of this model, one in which the learning agent and the mentor have identical abilities, and one designed to deal with agents and mentors with different action sets. We illustrate the benefits of implicit imitation by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability and possible interactions, we briefly comment on extensions of the model that relax these restricitions.

Machine Learning,Artificial Intelligence

Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Generative Model

Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model

Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning

On Computation and Generalization of Generative Adversarial Imitation Learning.

Acgail: Imitation Learning About Multiple Intentions With Auxiliary Classifier Gans

Generative Adversarial Imitation Learning from Failed Experiences

GAILPG: Multi-Agent Policy Gradient with Generative Adversarial Imitation Learning

A Pragmatic Look at Deep Imitation Learning

Distributional generative adversarial imitation learning with reproducing kernel generalization

Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulation

Addressing reward bias in Adversarial Imitation Learning with neutral reward functions

CRIL: Continual Robot Imitation Learning via Generative and Prediction Model

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

Interpretable Generative Adversarial Imitation Learning

Generative predecessor models for sample-efficient imitation learning

Diffusion-Reward Adversarial Imitation Learning

Self-imitation guided goal-conditioned reinforcement learning

Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets

Latent Policies for Adversarial Imitation Learning

Accelerating Reinforcement Learning through Implicit Imitation