Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning

Se-Wook Yoo,Seung-Woo Seo

DOI: https://doi.org/10.1109/ICRA46639.2022.9811697

2022-06-20

Abstract:Many robotic tasks are composed of a lot of temporally correlated sub-tasks in a highly complex environment. It is important to discover situational intentions and proper actions by deliberating on temporal abstractions to solve problems effectively. To understand the intention separated from changing task dynamics, we extend an empowerment-based regularization technique to situations with multiple tasks based on the framework of a generative adversarial network. Under the multitask environments with unknown dynamics, we focus on learning a reward and policy from the unlabeled expert examples. In this study, we define situational empowerment as the maximum of mutual information representing how an action conditioned on both a certain state and sub-task affects the future. Our proposed method derives the variational lower bound of the situational mutual information to optimize it. We simultaneously learn the transferable multi-task reward function and policy by adding an induced term to the objective function. By doing so, the multi-task reward function helps to learn a robust policy for environmental change. We validate the advantages of our approach on multi-task learning and multi-task transfer learning. We demonstrate our proposed method has the robustness of both randomness and changing task dynamics. Finally, we prove that our method has significantly better performance and data efficiency than existing imitation learning methods on various benchmarks.

Machine Learning,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to learn transferable multi - task reward functions and policies from unlabeled expert examples in a complex multi - task environment. Specifically, the author focuses on learning reward functions separated from environmental dynamics by introducing sub - task variables in a multi - task environment with unknown dynamics, and simultaneously learning robust policies that can adapt to environmental changes. To achieve this goal, the author proposes a method based on variational inverse reinforcement learning, namely Situational Empowerment - based Adversarial Inverse Reinforcement Learning (SEAIRL). This method learns situational empowerment by maximizing conditional mutual information, thereby optimizing the learning process of reward functions and policies. In addition, the author also demonstrates the advantages of their proposed method in multi - task learning and multi - task transfer learning, especially in handling randomness and changing task dynamics.