Abstract:In this paper, we consider a transfer reinforcement learning problem involving agents with different action spaces. Specifically, for any new unseen task, the goal is to use a successful demonstration of this task by an expert agent in its action space to enable a learner agent learn an optimal policy in its own different action space with fewer samples than those required if the learner was learning on its own. Existing transfer learning methods across different action spaces either require handcrafted mappings between those action spaces provided by human experts, which can induce bias in the learning procedure, or require the expert agent to share its policy parameters with the learner agent, which does not generalize well to unseen tasks. In this work, we propose a method that learns a subgoal mapping between the expert agent policy and the learner agent policy. Since the expert agent and the learner agent have different action spaces, their optimal policies can have different subgoal trajectories. We learn this subgoal mapping by training a Long Short Term Memory (LSTM) network for a distribution of tasks and then use this mapping to predict the learner subgoal sequence for unseen tasks, thereby improving the speed of learning by biasing the agent's policy towards the predicted learner subgoal sequence. Through numerical experiments, we demonstrate that the proposed learning scheme can effectively find the subgoal mapping underlying the given distribution of tasks. Moreover, letting the learner agent imitate the expert agent's policy with the learnt subgoal mapping can significantly improve the sample efficiency and training time of the learner agent in unseen new tasks.

Transfer with Action Embeddings for Deep Reinforcement Learning

Learning Action-Transferable Policy with Action Embedding

Efficient Deep Reinforcement Learning Through Policy Transfer.

Transfer Reinforcement Learning in Heterogeneous Action Spaces using Subgoal Mapping

A Transfer Approach Using Graph Neural Networks in Deep Reinforcement Learning

Efficient Deep Reinforcement Learning Via Adaptive Policy Transfer

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Transferring knowledge from human-demonstration trajectories to reinforcement learning

Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch

Introspective Action Advising for Interpretable Transfer Learning

Simple Emergent Action Representations from Multi-Task Policy Training

Deep Model Transferability from Attribution Maps

Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning

DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spaces

Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment

Shaping in Reinforcement Learning by Knowledge Transferred from Human-Demonstrations of a Simple Similar Task.

Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

Stochastic Ensemble Policy Transfer

DGTRL: Deep graph transfer reinforcement learning method based on fusion of knowledge and data

Multi-Agent Policy Transfer Via Task Relationship Modeling.